Data processing device, data processing method, program, and electronic device

ABSTRACT

There is provided an imaging system. The imaging system comprising a multispectral camera configured to capture a multispectral image of an object, an RGB camera configured to capture a color image of the object, at least one storage device configured to store spectrum information for each of a plurality of labeled objects, and processing circuitry. The processing circuitry is configured to determine, based on the captured multispectral image, spectrum information associated with the object, associate, based at least in part, on the spectrum information associated with the object and the stored spectrum information for each of the plurality of objects, a label with the color image of the object, and store, on the at least one storage device, the color image and the associated label as training data.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit under 35 U.S.C. § 120 as adivisional application of U.S. application Ser. No. 16/465,178, filed onMay 30, 2019, now U.S. Pat. No. 11,120,287, which claims the benefitunder 35 U.S.C. § 371 as a U.S. National Stage Entry of InternationalApplication No. PCT/JP2017/044631, filed in the Japanese Patent Officeas a Receiving Office on Dec. 12, 2017, which claims priority toJapanese Patent Application Number JP2016-241355, filed in the JapanesePatent Office on Dec. 13, 2016, each of which applications is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to a data processing device, a dataprocessing method, a program, and an electronic device, and inparticular, relates to a data processing device, a data processingmethod, a program, and an electronic device, in which labelingprocessing can be automated with respect to an object.

BACKGROUND ART

In the related art, a harvest prediction device is proposed in which thetypes of crops are recognized, and a yield amount is predicted withrespect to each of the crops, from an image in which various cropscultivated in a wide region are photographed (for example, refer to PTL1).

In addition, an imaging element detecting light in a predeterminednarrow wavelength band (a narrow band) (hereinafter, also referred to asnarrow band light) by using a plasmon filter is proposed (for example,refer to PTL 2).

CITATION LIST Patent Literature

[PTL 1]

JP 2003-6612A

[PTL 2]

JP 2010-165718A

SUMMARY Technical Problem

However, as described above, in object recognition processing ofrecognizing an object from an image, such as recognition of the type ofcrop photographed in the image, in general, it is necessary to performmachine learning in advance by using a large quantity of training data.In the related art, for example, the training data is generated byvisually determining and labeling the object, and thus, enormousman-hour is necessary for preparing a large quantity of training data.

The present disclosure has been made in consideration of thecircumstances described above, and is capable of automating the labelingof an object.

Solution to Problem

According to the present disclosure, there is provided an imagingsystem, comprising a multispectral camera configured to capture amultispectral image of an object, an RGB camera configured to capture acolor image of the object, at least one storage device configured tostore spectrum information for each of a plurality of labeled objects,and processing circuitry. The processing circuitry is configured todetermine, based on the captured multispectral image, spectruminformation associated with the object, associate, based at least inpart, on the spectrum information associated with the object and thestored spectrum information for each of the plurality of objects, alabel with the color image of the object, and store, on the at least onestorage device, the color image and the associated label as trainingdata.

Further according to the present disclosure, there is provided an objectclassification system, comprising at least one storage device configuredto store a trained object classifier and processing circuitry. Theprocessing circuitry is configured to classify an object in a receivedcolor image using the trained object classifier,

-   -   determine based, at least in part, on the classification of the        object in the received color image and spectrum information        associated with the object, an evaluation index value for a        characteristic of the object, and output on a display, an        indication of the evaluation index value.

Further according to the present disclosure, there is provided a methodof generating training data for training an object classifier. Themethod comprises receiving a multispectral image of an object capturedby a multispectral camera and a color image of the object captured by anRGB camera, and determining, based on the multispectral image of anobject, spectrum information associated with the object. The methodfurther comprises associating, based at least in part, on the spectruminformation associated with the object and stored spectrum informationfor each of a plurality of objects, a label with the color image of theobject, and storing, on at least one storage device, the color image andthe associated label as training data.

Further according to the present disclosure, there is provided a methodof classifying an object. The method comprises classifying an object ina received color image using a trained object classifier, determiningbased, at least in part, on the classification of the object in thereceived color image and spectrum information associated with theobject, an evaluation index value for a characteristic of the object,and outputting on a display, an indication of the evaluation indexvalue.

Advantageous Effects of Invention

According to one aspect of the present disclosure, it is possible toautomate the labeling of an object.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an embodiment of an imagingdevice to which the present technology is applied.

FIG. 2 is a block diagram illustrating a configuration example of acircuit of an imaging element.

FIG. 3 is a sectional view schematically illustrating configurationexample of a first embodiment of the imaging element.

FIG. 4 is a diagram illustrating a configuration example of a plasmonfilter having a pore array structure.

FIG. 5 is a graph illustrating a dispersion relationship of a frontplasmon.

FIG. 6 is a graph illustrating a first example of spectrum informationof the plasmon filter having the pore array structure.

FIG. 7 is a graph illustrating a second example of the spectruminformation of the plasmon filter having the pore array structure.

FIG. 8 is a graph illustrating a plasmon mode and a waveguide mode.

FIG. 9 is a graph illustrating an example of propagation characteristicsof the front plasmon.

FIGS. 10A and 10B are diagrams illustrating another configurationexample of the plasmon filter having the pore array structure.

FIG. 11 is a diagram illustrating a configuration example of a plasmonfilter having a two-layer structure.

FIGS. 12A and 12B are diagrams illustrating a configuration example of aplasmon filter having a dot array structure.

FIG. 13 is a graph illustrating an example of spectrum information ofthe plasmon filter having the dot array structure.

FIG. 14 is a diagram illustrating configuration example of a plasmonfilter using GMR.

FIG. 15 is a graph illustrating an example of spectrum information ofthe plasmon filter using GMR.

FIG. 16 is a sectional view schematically illustrating a configurationexample of a second embodiment of the imaging element.

FIG. 17 is a diagram schematically illustrating an aspect of occurrenceof flare of the imaging device.

FIG. 18 is a diagram for describing a flare reducing method of theimaging device.

FIG. 19 is a graph illustrating a first example of spectrum informationof a narrow band filter and a transmission filter.

FIG. 20 is a graph illustrating a second example of the spectruminformation of the narrow band filter and the transmission filter.

FIG. 21 is a graph illustrating a third example of the spectruminformation of the narrow band filter and the transmission filter.

FIG. 22 is a sectional view schematically illustrating a configurationexample of a third embodiment of the imaging element.

FIG. 23 is a block diagram illustrating a configuration example of anembodiment of a training data generating system to which the presenttechnology is applied.

FIG. 24 is a block diagram illustrating a configuration example of anobject recognition system.

FIG. 25 is a flowchart describing training data generating processing.

FIG. 26 is a block diagram illustrating a configuration example of anembodiment of an evaluation index presenting system to which the presenttechnology is applied.

FIGS. 27A and 27B are diagrams describing usage examples of aninformation processing terminal on which the evaluation index presentingsystem is mounted.

FIG. 28 is a flowchart describing evaluation index presentingprocessing.

FIG. 29 is a block diagram illustrating a configuration example of anembodiment of a computer.

FIGS. 30A to 30C are diagrams illustrating outlines of a configurationexample of a laminated solid imaging device to which the presenttechnology can be applied.

FIG. 31 is a diagram illustrating an application example of the presenttechnology.

FIG. 32 is a diagram illustrating an example of a detection band in acase where the tastiness or the freshness of food is detected.

FIG. 33 is a diagram illustrating an example of a detection band in acase where a sugar content or the moisture of fruit is detected.

FIG. 34 is a diagram illustrating an example of a detection band in acase where plastic is sorted.

FIG. 35 is a diagram illustrating an example of a schematicconfiguration of an endoscopic surgery system.

FIG. 36 is a block diagram illustrating an example of a functionalconfiguration of a camera head and CCU.

FIG. 37 is a block diagram illustrating an example of a schematicconfiguration of a vehicle control system.

FIG. 38 is an explanatory diagram describing an example of a dispositionposition of an outdoor information detecting unit and an imaging unit.

DESCRIPTION OF EMBODIMENTS

Hereinafter, aspects for carrying out the disclosure (hereinafter,referred to as an “embodiment”) will be described in detail by using thedrawings. Furthermore, the embodiments will be described in thefollowing sequence.

-   -   1. Embodiment of Imaging Device    -   2. Usage Example of Multispectral Image    -   3. Modification Example    -   4. Application Example

1. Embodiment of Imaging Device

First, an embodiment of an imaging device of the present technology willbe described with reference to FIGS. 1 to 22 .

<Configuration Example of Imaging Device>

FIG. 1 is a block diagram illustrating an embodiment of an imagingdevice, which is one type of electronic devices to which the presenttechnology is applied.

An imaging device 10 of FIG. 1 , for example, is formed of a digitalcamera which is capable of imaging both of a still image and a movingimage. In addition, the imaging device 10, for example, is formed of amultispectral camera which is capable of detecting light(multi-spectrum) of four or more wavelength bands (four or more bands)greater than three wavelength bands (three bands) of the related art ofR (red), G (green), and B (blue) or Y (yellow), M (magenta), and C(cyan), based on three primary colors or a color-matching function.

The imaging device 10 includes an optical system 11, an imaging element12, a memory 13, a signal processing unit 14, an output unit 15, and acontrol unit 16.

The optical system 11, for example, includes a zoom lens, a focus lens,a diaphragm, and the like, which are not illustrated, and allows lightfrom the outside to be incident on the imaging element 12. In addition,as necessary, various filters such as a polarization filter are disposedon the optical system 11.

The imaging element 12, for example, is formed of a complementary metaloxide semiconductor (CMOS) image sensor. The imaging element 12 receivesincident light from the optical system 11, and performs photoelectricconversion, and thus, outputs image data corresponding to the incidentlight.

The memory 13 temporarily stores the image data which is output from theimaging element 12.

The signal processing unit 14 performs signal processing (for example,processing such as elimination of a noise and adjustment of a whitebalance) using the image data stored in the memory 13, and thus,supplies the image data to the output unit 15.

The output unit 15 outputs the image data from the signal processingunit 14. For example, the output unit 15 includes a display (notillustrated) configured of a liquid crystal or the like, and displays aspectrum (an image) corresponding to the image data from the signalprocessing unit 14 as a so-called through image. For example, the outputunit 15 includes a driver (not illustrated) driving a recording mediumsuch as a semiconductor memory, a magnetic disk, and an optical disk,and records the image data from the signal processing unit 14 in arecording medium. For example, the output unit 15 functions as acommunication interface for performing communication with respect to anexternal device (not illustrated), and transmits the image data from thesignal processing unit 14 to the external device in a wireless manner ora wired manner.

The control unit 16 controls each of the units of the imaging device 10according to an operation or the like of a user.

<Configuration Example of Circuit of Imaging Element>

FIG. 2 is a block diagram illustrating a configuration example of acircuit of the imaging element 12 of FIG. 1 .

The imaging element 12 includes a pixel array 31, a row scanning circuit32, a phase locked loop (PLL) 33, a digital analog converter (DAC) 34, acolumn analog digital converter (ADC) circuit 35, a column scanningcircuit 36, and a sense amplifier 37.

A plurality of pixels 51 are two-dimensionally arranged in the pixelarray 31.

The pixel 51 includes a horizontal signal line H which is connected tothe row scanning circuit 32, a photodiode 61 which is disposed in eachpoint where the photodiode 61 intersects with a perpendicular signalline V connected to the column ADC circuit 35, and performsphotoelectric conversion, and several types of transistors for readingout an accumulated signal. That is, the pixel 51, as enlargedlyillustrated on the right side of FIG. 2 , includes the photodiode 61, atransfer transistor 62, a floating diffusion 63, an amplificationtransistor 64, a selection transistor 65, and a reset transistor 66.

An electric charge accumulated in the photodiode 61 is transferred tothe floating diffusion 63 through the transfer transistor 62. Thefloating diffusion 63 is connected to a gate of the amplificationtransistor 64. In a case where the pixel 51 is a target from which asignal is read out, the selection transistor 65 is turned on from therow scanning circuit 32 through the horizontal signal line H, and theamplification transistor 64 is subjected to source follower drivingaccording to the signal of the selected pixel 51, and thus, the signalis read out to the perpendicular signal line V as a pixel signalcorresponding to an accumulation electric charge amount of the electriccharge accumulated in the photodiode 61. In addition, the pixel signalis reset by turning on the reset transistor 66.

The row scanning circuit 32 sequentially outputs a driving (for example,transferring, selecting, resetting, or the like) signal for driving thepixel 51 of the pixel array 31 for each row.

The PLL 33 generates and outputs a clock signal of a predeterminedfrequency which is necessary for driving each of the units of theimaging element 12, on the basis of the clock signal supplied from theoutside.

The DAC 34 generates and outputs a lamp signal in the shape of beingreturned to a predetermined voltage value after a voltage drops from apredetermined voltage value at a certain slope (in the shape ofapproximately a saw).

The column ADC circuit 35 includes a comparator 71 and a counter 72 asmany as the number corresponding to the number of columns of the pixel51 of the pixel array 31, extracts a signal level from the pixel signaloutput from the pixel 51 by a correlated double sampling (CDS)operation, and outputs pixel data. That is, the comparator 71 comparesthe lamp signal supplied from the DAC 34 with the pixel signal (abrightness value) output from the pixel 51, and supplies a comparisonresult signal obtained as the result thereof to the counter 72. Then,the counter 72 counts a counter clock signal of a predeterminedfrequency according to the comparison result signal output from thecomparator 71, and thus, the pixel signal is subjected to A/Dconversion.

The column scanning circuit 36 sequentially supplies a signal ofoutputting the pixel data to the counter 72 of the column ADC circuit 35at a predetermined timing.

The sense amplifier 37 amplifies the pixel data which is supplied fromthe column ADC circuit 35, and outputs the pixel data to the outside ofthe imaging element 12.

<First Embodiment of Imaging Element>

FIG. 3 schematically illustrates a configuration example of a sectionalsurface of an imaging element 12A, which is a first embodiment of theimaging element 12 of FIG. 1 . FIG. 3 illustrates sectional surfaces offour pixels of a pixel 51-1 to a pixel 51-4 of the imaging element 12.Furthermore, hereinafter, in a case where it is not necessary todistinguish the pixel 51-1 to the pixel 51-4 from each other, the pixelwill be simply referred to as the pixel 51.

An on-chip microlens 101, an interlayer film 102, a narrow band filterlayer 103, an interlayer film 104, a photoelectric conversion elementlayer 105, and a signal wiring layer 106 are laminated in each of thepixels 51, in this order from the above. That is, the imaging element 12is formed of a back-side illumination type CMOS image sensor in whichthe photoelectric conversion element layer 105 is disposed on anincident side of light from the signal wiring layer 106.

The on-chip microlens 101 is an optical element for condensing lightinto the photoelectric conversion element layer 105 of each of thepixels 51.

The interlayer film 102 and the interlayer film 104 are formed of adielectric body such as SiO2. As described below, it is desirable thatdielectric constants of the interlayer film 102 and the interlayer film104 are as low as possible.

In the narrow band filter layer 103, a narrow band filter NB, which isan optical filter transmitting narrow band light in a predeterminednarrow wavelength band (a narrow band), is disposed in each of thepixels 51. For example, a plasmon filter using front plasmon, which isone type of metal thin film filters using a thin film formed of a metalsuch as aluminum, is used in the narrow band filter NB. In addition, atransmission band of the narrow band filter NB is set for each of thepixels 51. The type (the number of bands) of the transmission band ofthe narrow band filter NB is arbitrary, and for example, the number ofbands is set to be greater than or equal to 4.

Here, the narrow band, for example, is a wavelength band which isnarrower than a transmission band of a color filter of the related artof red (R), green (G), and blue (B) or yellow (Y), magenta (M), and cyan(C), based on three primary colors or a color-matching function. Inaddition, hereinafter, a pixel receiving the narrow band lighttransmitted through the narrow band filter NB will be referred to as amultispectral pixel or a MS pixel.

The photoelectric conversion element layer 105, for example, includesthe photodiode 61 or the like of FIG. 2 , receives the light transmittedthrough the narrow band filter layer 103 (the narrow band filter NB)(the narrow band light), and converts the received light into anelectric charge. In addition, the photoelectric conversion element layer105 is configured such that the pixels 51 are electrically separatedfrom each other by an element separating layer.

Wiring or the like for reading the electric charge which is accumulatedin the photoelectric conversion element layer 105 is disposed on thesignal wiring layer 106.

<Plasmon Filter>

Next, the plasmon filter which can be used in the narrow band filter NBwill be described with reference to FIGS. 4 to 15.

FIG. 4 illustrates a configuration example of a plasmon filter 121Ahaving a pore array structure.

The plasmon filter 121A is configured of a plasmon resonator in whichholes 132A are arranged in a metal thin film (hereinafter, referred toas a conductor thin film) 131A in the shape of a honeycomb.

Each of the holes 132A penetrates through the conductor thin film 131A,and functions as a waveguide. In general, the waveguide has a cutofffrequency and a cutoff wavelength which are determined according to ashape such as a length of a side or a diameter, and has properties ofnot allowing light of a frequency less than or equal to the cutofffrequency (a wavelength less than or equal to the cutoff wavelength) topropagate. A cutoff wavelength of the hole 132A mainly depends on anopening diameter D1, and the cutoff wavelength shortens as the openingdiameter D1 decreases. Furthermore, the opening diameter D1 is set to avalue which is smaller than the wavelength of the transmitted light.

On the other hand, in a case where light is incident on the conductorthin film 131A in which holes 132A are periodically formed during ashort period less than or equal to the wavelength of the light, aphenomenon occurs in which light at a wavelength which is longer thanthe cutoff wavelength of the hole 132A is transmitted. Such a phenomenonwill be referred to as an abnormal transmission phenomenon of theplasmon. Such a phenomenon occurs due to the excitation of front plasmonon a boundary between the conductor thin film 131A and the interlayerfilm 102, which is an upper layer of the conductor thin film 131A.

Here, occurrence conditions of the abnormal transmission phenomenon ofthe plasmon (a front plasmon resonance) will be described with referenceto FIG. 5 .

FIG. 5 is a graph illustrating a dispersion relationship of the frontplasmon. In the graph, a horizontal axis represents an angular wavenumber vector k, and a vertical axis represents an angular frequency ω.ωp represents a plasma frequency of the conductor thin film 131A. ωsprepresents a front plasma frequency on a boundary surface between theinterlayer film 102 and the conductor thin film 131A, and is representedby formula (1) described below.

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 1} \right\rbrack & \; \\{\omega_{sp} = \frac{\omega_{p}}{\sqrt{1 + ɛ_{d}}}} & (1)\end{matrix}$

εd represents a dielectric constant of a dielectric body configuring theinterlayer film 102.

According to formula (1), the front plasma frequency ωsp increases asthe plasma frequency ωp increases. In addition, the front plasmafrequency ωsp increases as the dielectric constant εd decreases.

A line L1 represents a dispersion relationship of the light (a writeline), and is represented by formula (2) described below.

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 2} \right\rbrack & \; \\{\omega = {\frac{c}{\sqrt{ɛ_{d}}}k}} & (2)\end{matrix}$

c represents a light speed.

A line L2 represents a dispersion relationship of the front plasmon, andis represented by formula (3) described below.

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 3} \right\rbrack & \; \\{\omega = {{ck}\sqrt{\frac{ɛ_{m} + ɛ_{d}}{ɛ_{m}ɛ_{d}}}}} & (3)\end{matrix}$

εm represents a dielectric constant of the conductor thin film 131A.

The dispersion relationship of the front plasmon represented by the lineL2 is close to the write line represented by the line L1 in a rangewhere the angular wave number vector k is small, and is close to thefront plasma frequency ωsp as the angular wave number vector kincreases.

Then, when formula (4) described below is established, the abnormaltransmission phenomenon of the plasmon occurs.

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 4} \right\rbrack & \; \\{{{Re}\left\lbrack {\frac{\omega_{sp}}{c}\sqrt{\frac{ɛ_{m}ɛ_{d}}{ɛ_{m} + ɛ_{d}}}} \right\rbrack} = {{{\frac{2\pi}{\lambda}\sin\;\theta} + {iG}_{x} + {jG}_{y}}}} & (4)\end{matrix}$

λ represents the wavelength of the incident light. θ represents anincident angle of the incident light. Gx and Gy are represented byformula (5) described below.|Gx|=|Gy|=2π/a0  (5)

a0 represents a lattice constant of a pore array structure formed of thehole 132A of the conductor thin film 131A.

In formula (4), the left member represents an angular wave number vectorof the front plasmon, and the right member represents an angular wavenumber vector of the conductor thin film 131A during a pore arrayperiod. Accordingly, when the angular wave number vector of the frontplasmon is identical to the angular wave number vector of the conductorthin film 131A during the pore array period, the abnormal transmissionphenomenon of the plasmon occurs. Then, at this time, the value of λ isa resonance wavelength of the plasmon (the transmission wavelength ofthe plasmon filter 121A).

Furthermore, in formula (4), the angular wave number vector of the frontplasmon in the left member is determined according to the dielectricconstant εm of the conductor thin film 131A and the dielectric constantεd of the interlayer film 102. On the other hand, the angular wavenumber vector during the pore array period in the right member isdetermined according to the incident angle θ of the light and a pitch (ahole pitch) P1 between the adjacent holes 132A of the conductor thinfilm 131A. Accordingly, the resonance wavelength and the resonancefrequency of the plasmon are determined according to the dielectricconstant εm of the conductor thin film 131A, the dielectric constant εdof the interlayer film 102, the incident angle θ of the light, and thehole pitch P1. Furthermore, in a case where the incident angle of thelight is 0°, the resonance wavelength and the resonance frequency of theplasmon are determined according to the dielectric constant εm of theconductor thin film 131A, the dielectric constant εd of the interlayerfilm 102, and the hole pitch P1.

Accordingly, the transmission band of the plasmon filter 121A (theresonance wavelength of the plasmon) is changed according to a materialand a film thickness of the conductor thin film 131A, a material and afilm thickness of the interlayer film 102, a pattern period of the porearray (for example, the opening diameter D1 and the hole pitch P1 of thehole 132A), and the like. In particular, in a case where the materialand the film thickness of the conductor thin film 131A and theinterlayer film 102 are determined, the transmission band of the plasmonfilter 121A is changed according to the pattern period of the porearray, in particular, the hole pitch P1. That is, the transmission bandof the plasmon filter 121A is shifted to a short wavelength side as thehole pitch P1 narrows, and the transmission band of the plasmon filter121A is shifted to a long wavelength side as the hole pitch P1 widens.

FIG. 6 is a graph illustrating an example of spectral characteristics ofthe plasmon filter 121A in a case where the hole pitch P1 is changed. Inthe graph, a horizontal axis represents a wavelength (the unit is nm),and a vertical axis represents sensitivity (the unit is an arbitraryunit). A line L11 represents spectral characteristics in a case wherethe hole pitch P1 is set to 250 nm, a line L12 represents spectralcharacteristics in a case where the hole pitch P1 is set to 325 nm, anda line L13 represents spectral characteristics in a case where the holepitch P1 is set to 500 nm.

In a case where the hole pitch P1 is set to 250 nm, the plasmon filter121A mainly transmits light in a wavelength band of a blue color. In acase where the hole pitch P1 is set to 325 nm, the plasmon filter 121Amainly transmits light in a wavelength band of a green color. In a casewhere the hole pitch P1 is set to 500 nm, the plasmon filter 121A mainlytransmits light in a wavelength band of a red color. However, in a casewhere the hole pitch P1 is set to 500 nm, the plasmon filter 121Atransmits a great amount of light in a low wavelength band of a redcolor according to a waveguide mode described below.

FIG. 7 is a graph illustrating another example of the spectralcharacteristics of the plasmon filter 121A in a case where the holepitch P1 is changed. In the graph, a horizontal axis represents awavelength (the unit is nm), and a vertical axis represents sensitivity(the unit is an arbitrary unit). This example illustrates an example ofspectral characteristics of sixteen types of plasmon filters 121A in acase where the hole pitch P1 is changed by being divided by 25 nm from250 nm to 625 nm.

Furthermore, the transmittance of the plasmon filter 121A is mainlydetermined according to the opening diameter D1 of the hole 132A. Thetransmittance increases as the opening diameter D1 increases, but colormixture easily occurs. In general, it is desirable that the openingdiameter D1 is set such that an opening rate is 50% to 60% of the holepitch P1.

In addition, as described above, each of the holes 132A of the plasmonfilter 121A functions as a waveguide. Accordingly, in the spectralcharacteristics, there is a case where not only a wavelength componenttransmitted by the front plasmon resonance (a wavelength component in aplasmon mode) but also a wavelength component transmitted through thehole 132A (the waveguide) (a wavelength component in a waveguide mode)increases, according to a pattern of the pore array of the plasmonfilter 121A.

For a given hole pitch P1 of the plasmon filter there is a range ofdesirable thicknesses of the plasmon filter to maximize lighttransmittance of the filter for those wavelengths that are transmitted.For instance, a range of desirable thicknesses of the plasmon filter mayrange between 20% and 80% of the size of the hole pitch P1, or between30% and 70% of the size of the hole pitch, or between 40% and 60% of thesize of the hole pitch.

For example, in a case where the plasmon filter is formed from Aluminum,a desirable range of thicknesses of the plasmon filter for a 350 nm holepitch is between 100 nm and 300 nm, with a preferred thickness of 200nm. For an Aluminum plasmon filter with a 550 nm hole pitch, a desirablerange of thicknesses of the plasmon filter is between 200 nm and 400 nm,with a preferred thickness of 300 nm.

For a given peak transmission wavelength of the plasmon filter there isa range of desirable thicknesses of the plasmon filter to maximize lighttransmittance of the filter for those wavelengths that are transmitted.For instance, a range of desirable thicknesses of the plasmon filter mayrange between 10% and 60% of the peak transmission wavelength, orbetween 20% and 50% of the peak transmission wavelength, or between 30%and 40% of the peak transmission wavelength.

For example, in a case where the plasmon filter is formed from Aluminum,a desirable range of thicknesses of the plasmon filter when desirable apeak transmission wavelength of 580 nm is between 100 nm and 300 nm,with a preferred thickness of 200 nm. For an Aluminum plasmon filterwith a peak transmission wavelength of 700 nm, a desirable range ofthicknesses of the plasmon filter is between 150 nm and 350 nm, with apreferred thickness of 250 nm.

FIG. 8 illustrates the spectral characteristics of the plasmon filter121A in a case where the hole pitch P1 is set to 500 nm, as with thespectral characteristics represented by the line L13 of FIG. 6 . In thisexample, a wavelength side which is longer than the cutoff wavelength inthe vicinity of 630 nm is the wavelength component in the plasmon mode,and a wavelength side which is shorter than the cutoff wavelength is thewavelength component in the waveguide mode.

As described above, the cutoff wavelength mainly depends on the openingdiameter D1 of the hole 132A, and the cutoff wavelength decreases as theopening diameter D1 decreases. Then, wavelength resolutioncharacteristics of the plasmon filter 121A are improved as a differencebetween the cutoff wavelength and the peak wavelength in the plasmonmode increases.

In addition, as described above, the front plasma frequency ωsp of theconductor thin film 131A increases as the plasma frequency ωp of theconductor thin film 131A increases. In addition, the front plasmafrequency ωsp increases as the dielectric constant εd of the interlayerfilm 102 decreases. Then, it is possible to set the resonance frequencyof the plasmon to be higher as the front plasma frequency ωsp increases,and to set the transmission band of the plasmon filter 121A (theresonance wavelength of the plasmon) to a shorter wavelength band.

Accordingly, in a case where a metal having a smaller plasma frequencyωp is used in the conductor thin film 131A, it is possible to set thetransmission band of the plasmon filter 121A to a shorter wavelengthband. For example, aluminum, silver, gold, or the like is preferable asthe metal. Here, in a case where the transmission band is set to a longwavelength band of infrared light or the like, copper or the like canalso be used.

In addition, in a case where a dielectric body having a small dielectricconstant εd is used in the interlayer film 102, it is possible to setthe transmission band of the plasmon filter 121A to a shorter wavelengthband. For example, SiO2, Low-K, or the like is preferable as thedielectric body.

In addition, FIG. 9 is a graph illustrating propagation characteristicsof the front plasmon on an interface between conductor thin film 131Aand the interlayer film 102 in a case where aluminum is used in theconductor thin film 131A, and SiO2 is used in the interlayer film 102.In the graph, a horizontal axis represents the wavelength of the light(the unit is nm), and a vertical axis represents a propagation distance(the unit is μm). In addition, a line L21 represents propagationcharacteristics in an interface direction, a line L22 representspropagation characteristics in a depth direction of the interlayer film102 (a direction perpendicular to the interface), and a line L23represents propagation characteristics in a depth direction of theconductor thin film 131A (a direction perpendicular to the interface).

A propagation distance ΛSPP (λ) in a depth direction of the frontplasmon is represented by formula (6) described below.

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 5} \right\rbrack & \; \\{{{\Lambda_{SPP}(\lambda)} \equiv \frac{4\pi\; k_{SPP}}{\lambda}} = {\frac{4\pi}{\lambda}{{Im}\left\lbrack \frac{ɛ_{m}ɛ_{d}}{ɛ_{m} + ɛ_{d}} \right\rbrack}}} & (6)\end{matrix}$

kSPP represents an absorption coefficient of a substance propagated bythe front plasmon. εm (λ) represents a dielectric constant of theconductor thin film 131A with respect to light at a wavelength of λ. εd(λ) represents a dielectric constant of the interlayer film 102 withrespect to light at the wavelength of λ.

Accordingly, as illustrated in FIG. 9 , front plasmon with respect tolight at a wavelength of 400 nm propagates in the depth direction from afront surface of the interlayer film 102 formed of SiO2 to approximately100 nm. Accordingly, the thickness of the interlayer film 102 is set tobe greater than or equal to 100 nm, and thus, the front plasmon on theinterface between the interlayer film 102 and the conductor thin film131A is prevented from being affected by a substance laminated on asurface of the interlayer film 102 on a side opposite to the conductorthin film 131A.

In addition, front plasmon with respect to light at a wavelength of 400nm propagates in the depth direction from a front surface of theconductor thin film 131A formed of aluminum to approximately 10 nm.Accordingly, the thickness of the conductor thin film 131A is set to begreater than or equal to 10 nm, and thus, the front plasmon on theinterface between the interlayer film 102 and the conductor thin film131A is prevented from being affected by the interlayer film 104.

<Other Examples of Plasmon Filter>

Next, other examples of the plasmon filter will be described withreference to FIGS. 10A to 15 .

A plasmon filter 121B of FIG. 10A is configured of a plasmon resonatorin which holes 132B are arranged in a conductor thin film 131B in theshape of an orthogonal matrix. In the plasmon filter 121B, for example,a transmission band is changed according to a pitch P2 between adjacentholes 132B.

In addition, in the plasmon resonator, it is not necessary that all ofthe holes penetrate through the conductor thin film, and even in a casewhere a part of the holes is configured as a non-through which does notpenetrate through the conductor thin film, the plasmon resonatorfunctions as a filter.

For example, in FIG. 10B, a plan view and a sectional view (a sectionalview taken along A-A′ of the plan view) of a plasmon filter 121Cconfigured of a plasmon resonator in which holes 132C formed of throughholes and holes 132C′ formed of non-through holes are arranged in theconductor thin film 131C in the shape of a honeycomb. That is, holes132C formed of through holes and holes 132C′ formed of non-through holesare periodically arranged in the plasmon filter 121C.

Further, a plasmon resonator of a single layer is basically used as theplasmon filter, and for example, the plasmon filter can be configured ofa two-layer plasmon resonator.

For example, a plasmon filter 121D illustrated in FIG. 11 is configuredof two layers of a plasmon filter 121D-1 and a plasmon filter 121D-2.The plasmon filter 121D-1 and the plasmon filter 121D-2 have a structurein which holes are arranged in the shape of a honeycomb, as with theplasmon resonator configuring the plasmon filter 121A of FIG. 4 .

In addition, it is preferable that an interval D2 between the plasmonfilter 121D-1 and the plasmon filter 121D-2 is approximately ¼ of a peakwavelength of a transmission band. In addition, in consideration of thefreedom in design, it is preferable that the interval D2 is less than orequal to ½ of the peak wavelength of the transmission band.

Furthermore, as with the plasmon filter 121D, the holes are arranged inthe same pattern in the plasmon filter 121D-1 and the plasmon filter121D-2, and for example, the holes may be arranged in patterns similarto each other in a two-layer plasmon resonator structure. In addition,in the two-layer plasmon resonator structure, holes and dots may bearranged in a pattern in which a pore array structure and a dot arraystructure (described below) are inversed from each other. Further, theplasmon filter 121D has the two-layer structure, and is capable of beingmultilayered to be three or more layers.

In addition, in the above description, the configuration example of theplasmon filter using the plasmon resonator having the pore arraystructure has been described, but a plasmon resonator having a dot arraystructure may be adopted as the plasmon filter.

A plasmon filter having a dot array structure will be described withreference to FIGS. 12A and 12B.

A plasmon filter 121A′ of FIG. 12A has a structure which is negativelyand positively inversed with respect to the plasmon resonator of theplasmon filter 121A of FIG. 4 , that is, is configured of a plasmonresonator in which dots 133A are arranged in a dielectric layer 134A inthe shape of a honeycomb. A space between the respective dots 133A isfilled with the dielectric layer 134A.

The plasmon filter 121A′ absorbs light in a predetermined wavelengthband, and thus, is used as a complementary color filter. The wavelengthband of the light which is absorbed by the plasmon filter 121A′(hereinafter, referred to as an absorption band) is changed according toa pitch (hereinafter, referred to as a dot pitch) P3 between theadjacent dots 133A. In addition, a diameter D3 of the dot 133A isadjusted according to the dot pitch P3.

A plasmon filter 121B′ of FIG. 12B has a structure which is negativelyand positively inversed with respect to the plasmon resonator of theplasmon filter 121B of FIG. 10A, that is, is configured of a plasmonresonator structure in which dots 133B are arranged in a dielectriclayer 134B in the shape of an orthogonal matrix. A space between therespective dots 133B is filled with the dielectric layer 134B.

An absorption band of the plasmon filter 121B′ is changed according to adot pitch P4 or the like between the adjacent dots 133B. In addition, adiameter D3 of the dot 133B is adjusted according to the dot pitch P4.

FIG. 13 is a graph illustrating an example of spectral characteristicsin a case where the dot pitch P3 of the plasmon filter 121A′ of FIG. 12Ais changed. In the graph, a horizontal axis represents a wavelength (theunit is nm), and a vertical axis represents transmittance. A line L31represents spectral characteristics in a case where the dot pitch P3 isset to 300 nm, a line L32 represents spectral characteristics in a casewhere the dot pitch P3 is set to 400 nm, and a line L33 representsspectral characteristics in a case where the dot pitch P3 is set to 500nm.

As illustrated in the drawing, the absorption band of the plasmon filter121A′ is shifted to a short wavelength side as the dot pitch P3 narrows,and the absorption band of the plasmon filter 121A′ is shifted to a longwavelength side as the dot pitch P3 widens.

Furthermore, in both of the plasmon filters having the pore arraystructure and the dot array structure, the transmission band or theabsorption band can be adjusted by only adjusting the pitch between theholes or the dots in a planar direction. Accordingly, for example, thetransmission band or the absorption band can be individually set withrespect to each pixel by only adjusting the pitch between the holes orthe dots in a lithography process, and the filter can be multicoloredthrough a fewer process.

In addition, the thickness of the plasmon filter is approximately 100 nmto 500 nm, which is approximately similar to that of a color filter ofan organic material, and a process affinity is excellent.

In addition, a plasmon filter 151 using a guided mode resonant (GMR)illustrated in FIG. 14 can also be used in the narrow band filter NB.

A conductor layer 161, an SiO2 film 162, an SiN film 163, and an SiO2substrate 164 are laminated in the plasmon filter 151, in this orderfrom the above. The conductor layer 161, for example, is included in thenarrow band filter layer 103 of FIG. 3 , and the SiO2 film 162, the SiNfilm 163, and the SiO2 substrate 164, for example, are included in theinterlayer film 104 of FIG. 3 .

For example, rectangular conductor thin films 161A formed of aluminumare arranged in the conductor layer 161 such that long sides of theconductor thin films 161A are adjacent to each other at a predeterminedpitch P5. Then, a transmission band of the plasmon filter 151 is changedaccording to the pitch P5 or the like.

FIG. 15 is a graph illustrating an example of spectral characteristicsof the plasmon filter 151 in a case where the pitch P5 is changed. Inthe graph, a horizontal axis represents a wavelength (the unit is nm),and a vertical axis represents transmittance. This example illustratesan example of spectral characteristics in a case where the pitch P5 ischanged to six types of pitches by being divided by 40 nm from 280 nm to480 nm, and a width of a slit between the adjacent conductor thin films161A is set to be ¼ of the pitch P5. In addition, a waveform having theshortest peak wavelength of the transmission band represents spectralcharacteristics in a case where the pitch P5 is set to 280 nm, and thepeak wavelength elongates as the pitch P5 widens. That is, thetransmission band of the plasmon filter 151 is shifted to a shortwavelength side as the pitch P5 narrows, and the transmission band ofthe plasmon filter 151 is shifted to a long wavelength side as the pitchP5 widens.

The plasmon filter 151 using GMR has excellent affinity with respect toa color filter of an organic material, as with the plasmon filtershaving the pore array structure and the dot array structure describedabove.

<Second Embodiment of Imaging Element>

Next, a second embodiment of the imaging element 12 of FIG. 1 will bedescribed with reference to FIGS. 16 to 21 .

FIG. 16 schematically illustrates a configuration example of a sectionalsurface of an imaging element 12B which is the second embodiment of theimaging element 12. Furthermore, in the drawing, the same referencenumerals are applied to portions corresponding to the imaging element12A of FIG. 3 , and the description thereof will be suitably omitted.

The imaging element 12B is different from the imaging element 12A inthat a color filter layer 107 is laminated between the on-chip microlens101 and the interlayer film 102.

In the narrow band filter layer 103 of the imaging element 12B, thenarrow band filter NB is disposed in a part of the pixels 51 but not allof the pixels 51. The type of the transmission band of the narrow bandfilter NB (the number of bands) is arbitrary, and for example, thenumber of bands is set to be greater than or equal to 1.

In the color filter layer 107, a color filter is disposed in each of thepixels 51. For example, in the pixel 51 where the narrow band filter NBis not disposed, any one of a general red color filter R, a generalgreen color filter G, and a general blue color filter B (notillustrated) is disposed. Accordingly, for example, an R pixel in whichthe red color filter R is disposed, a G pixel in which the green colorfilter G is disposed, a B pixel in which the blue color filter isdisposed, and an MS pixel in which in which the narrow band filter NB isdisposed, are arranged in the pixel array 31.

In addition, in the pixel 51 where the narrow band filter NB isdisposed, a transmission filter P is disposed on the color filter layer107. As described below, the transmission filter P is configured of anoptical filter transmitting light in a wavelength band including thetransmission band of the narrow band filter NB of the same pixel 51 (alow pass filter, a high pass filter, or a band pass filter).

Furthermore, the color filter disposed on the color filter layer 107 maybe color filters of both of an organic material and an inorganicmaterial.

Examples of the color filter of the organic material include a dyeingand coloring color filter of a synthetic resin or natural protein, and acolor filter containing a dye using a pigment dye or a colorant dye.

Examples of the color filter of the inorganic material include materialssuch as TiO2, ZnS, SiN, MgF2, SiO2, and Low-k. In addition, for example,a method such as vapor deposition, sputtering, and chemical vapordeposition (CVD) film formation is used for forming the color filter ofthe inorganic material.

In addition, as described above with reference to FIG. 9 , theinterlayer film 102 is set to have a film thickness which is capable ofpreventing the influence of the color filter layer 107 on the frontplasmon on an interface between the interlayer film 102 and the narrowband filter layer 103.

Here, the occurrence of flare is suppressed by the transmission filter Pdisposed on the color filter layer 107. This will be described withreference to FIGS. 17 and 18 .

FIG. 17 schematically illustrates an aspect of the occurrence of theflare of the imaging device 10 using the imaging element 12A of FIG. 2in which the color filter layer 107 is not disposed.

In this example, the imaging element 12A is disposed on a semiconductorchip 203. Specifically, the semiconductor chip 203 is mounted on asubstrate 213, and is surrounded by seal glass 211 and a resin 212.Then, light transmitted through a lens 201, an IR cut filter 202, andthe seal glass 211, which are disposed in the optical system 11 of FIG.1 , is incident on the imaging element 12A.

Here, in a case where the narrow band filter NB of the narrow bandfilter layer 103 of the imaging element 12A is formed of a plasmonfilter, a conductor thin film formed of metal is formed in the plasmonfilter. The conductor thin film has a high reflection rate, and thus,light at a wavelength other than the transmission band is easilyreflected. Then, a part of the light reflected on the conductor thinfilm, for example, as illustrated in FIG. 17 , is reflected on the sealglass 211, the IR cut filter 202, or the lens 201, and is incident againon the imaging element 12A. The flare occurs due to the re-incidentlight. In particular, a plasmon filter using a pore array structure hasa low opening rate, and thus, the flare easily occurs.

In order to prevent the reflection light, for example, it is consideredthat an antireflection film formed of a metal or a material having ahigh dielectric constant, which is different from the conductor thinfilm, is used. However, in a case where the plasmon filter uses a frontplasmon resonance, and such an antireflection film is in contact withthe front surface of the conductor thin film, there is a possibilitythat the characteristics of the plasmon filter are degraded, and desiredcharacteristics are not obtained.

On the other hand, FIG. 18 schematically illustrates an aspect of theoccurrence of the flare of the imaging device 10 using the imagingelement 12B of FIG. 16 , in which the color filter layer 107 isdisposed. Furthermore, in the drawing, the same reference numerals areapplied to portions corresponding to those of FIG. 17 .

The example of FIG. 18 is different from the example of FIG. 17 in thata semiconductor chip 221 is disposed instead of the semiconductor chip203. The semiconductor chip 221 is different from the semiconductor chip203 in that the imaging element 12B is disposed instead of the imagingelement 12A.

As described above, in the imaging element 12B, the transmission filterP is disposed on an upper side from the narrow band filter NB (anincident side of light). Accordingly, the light incident on the imagingelement 12B is incident on the narrow band filter NB, in which apredetermined wavelength band is cutoff, by the transmission filter P,and thus, a light amount of the incident light with respect to thenarrow band filter NB is suppressed. As a result thereof, a light amountof the reflection light on the conductor thin film of the narrow bandfilter NB (the plasmon filter) is also reduced, and thus, the flare isreduced.

FIGS. 19 to 21 illustrate examples of the spectral characteristics ofthe narrow band filter NB and the spectral characteristics of thetransmission filter P disposed on the upper side of the narrow bandfilter NB. Furthermore, in the graphs of FIGS. 19 to 21 , a horizontalaxis represents a wavelength (the unit is nm), and a vertical axisrepresents sensitivity (the unit is an arbitrary unit).

In the FIG. 19 , a line L41 represents the spectral characteristics ofthe narrow band filter NB. A peak wavelength of the spectralcharacteristics of the narrow band filter NB is approximately in thevicinity of 430 nm. A line L42 represents the spectral characteristicsof a low pass type transmission filter P. A line L43 represents thespectral characteristics of a high pass type transmission filter P. Aline L44 represents the spectral characteristics of a band pass typetransmission filter P. The sensitivities of all of the transmissionfilters P are greater than the sensitivity of the narrow band filter NBin a predetermined wavelength band including the peak wavelength of thespectral characteristics of the narrow band filter NB. Accordingly, itis possible to reduce the light amount of the incident light which isincident on the narrow band filter NB without substantially attenuatingthe light in the transmission band of the narrow band filter NB, byusing any transmission filter P.

In FIG. 20 , a line L51 represents the spectral characteristics ofnarrow band filter NB. A peak wavelength of the spectral characteristicsof the narrow band filter NB is approximately in the vicinity of 530 nm.A line L52 represents the spectral characteristics of the low pass typetransmission filter P. A line L53 represents the spectralcharacteristics of the high pass type transmission filter P. A line L54represents the spectral characteristics of the band pass typetransmission filter P. The sensitivities of all of the transmissionfilters are greater than the sensitivity of the narrow band filter NB ina predetermined wavelength band including the peak wavelength of thespectral characteristics of the narrow band filter NB. Accordingly, itis possible to reduce the light amount of the incident light which isincident on the narrow band filter NB without substantially attenuatingthe light in the transmission band of the narrow band filter NB, byusing any transmission filter P.

In FIG. 21 , a line L61 represents the spectral characteristics ofnarrow band filter NB. A peak wavelength of the spectral characteristicsof the narrow band filter NB in a plasmon mode is approximately in thevicinity of 670 nm. A line L62 represents the spectral characteristicsof the low pass type transmission filter P. A line L63 represents thespectral characteristics of the high pass type transmission filter P. Aline L64 represents the spectral characteristics of the band pass typetransmission filter P. The sensitivities of all of the transmissionfilters are greater than the sensitivity of the narrow band filter NB ina predetermined wavelength band including the peak wavelength in theplasmon mode of greater than or equal to 630 nm, which is the cutoffwavelength of the spectral characteristics of the narrow band filter NB.Accordingly, it is possible to reduce the light amount of the incidentlight which is incident on the narrow band filter NB withoutsubstantially attenuating the light in the transmission band of thenarrow band filter NB in the plasmon mode, by using any transmissionfilter P. Here, using the high pass type transmission filter P or theband pass type transmission filter P is desirable as the characteristicsof a narrow band filter since light in a wavelength band of the narrowband filter NB in a waveguide mode can be cutoff.

Furthermore, in a case where the transmission band of the red colorfilter R, the green color filter G, or the blue color filter B includesa transmission band of the narrow band filter NB of a lower layer, suchfilters may be used in the transmission filter P.

In addition, in the example of FIG. 16 , an example is described inwhich the narrow band filter NB is disposed only in a part of the pixels51, and the narrow band filter NB is capable of being disposed in all ofthe pixels 51. In this case, in each of the pixels 51, the transmissionfilter P having a transmission band which includes the transmission bandof the narrow band filter NB of the pixel 51 may be disposed on thecolor filter layer 107.

Further, a combination of the colors of the color filters in the colorfilter layer 107 is not limited to the example described above, and canbe arbitrarily changed.

In addition, in a case where a countermeasure against the flaredescribed above is not necessary, for example, the transmission filter Pmay be disposed on an upper layer of the narrow band filter NB, or adummy filter transmitting light at all wavelengths may be disposed.

<Third Embodiment of Imaging Element>

Next, a third embodiment of the imaging element 12 of FIG. 1 will bedescribed with reference to FIG. 22 .

FIG. 22 schematically illustrates a configuration example of a sectionalsurface of an imaging element 12C, which is the third embodiment of theimaging element 12. Furthermore, in the drawing, the same referencenumerals are applied to portions corresponding to the imaging element12A of FIG. 3 , and the description thereof will be suitably omitted.

The imaging element 12C is different from the imaging element 12A inthat a filter layer 108 is disposed instead of the narrow band filterlayer 103. In addition, the imaging element 12C is different from theimaging element 12B of FIG. 16 in that the narrow band filter NB and thecolor filter (for example, the red color filter R, the green colorfilter G, and the blue color filter B) are disposed in the same filterlayer 108.

Accordingly, in a case where the R pixel, the G pixel, the B pixel, andthe MS pixel are arranged in the pixel array 31 of the imaging element12C, the color filter layer 107 can be omitted.

Furthermore, in a case where the color filter of the organic material isused, in order to prevent a damage or the like of the color filter dueto heat, for example, the narrow band filter NB is formed first, andfinal heat processing such as sinter processing is performed at a hightemperature, and then, the color filter is formed. On the other hand, ina case where the color filter of the inorganic material is used,basically, there is no necessity to restrict the formation sequencedescribed above.

In addition, in a case where the countermeasure against the flare isperformed as in the imaging element 12B of FIG. 16 , as with the imagingelement 12B, the color filter layer may be laminated between the on-chipmicrolens 101 and the interlayer film 102. In this case, in the pixel 51where the narrow band filter NB is disposed on the filter layer 108, thetransmission filter P described above is disposed on the color filterlayer. On the other hand, in the pixel 51 where the color filter isdisposed on the filter layer 108, a filter may be disposed on the colorfilter layer, or a dummy filter transmitting light in all wavelengths ora color filter of the same color as that of the filter layer 108 may bedisposed.

2. Usage Example of Multispectral Image

Next, processing performed by using an image output from the imagingelement 12 of FIG. 1 (hereinafter, referred to as a multispectral image)will be described with reference to FIGS. 23 to 29 .

<First Usage Example>

FIG. 23 illustrates a configuration example of a training datagenerating system performing processing of automatically generatingtraining data to be used for machine learning, as a first usage exampleof processing which is performed by using the multispectral image.

As illustrated in FIG. 23 , a training data generating system 301 isconfigured of a multispectral camera 311, an RGB camera 312, a storagedevice 313, and a training data generating processing device 314.

The multispectral camera 311 includes the imaging element 12 of FIG. 1 ,and supplies the multispectral image obtained by imaging an object (forexample, an apple, a face of a person, or the like), which is a targetgenerating the training data, to the training data generating processingdevice 314.

The RGB camera 312 is an imaging device which is capable of imaging acolor image (so-called an RGB image), images the same object as thatimaged by the multispectral camera 311, and supplies a color imageobtained as a result thereof to the training data generating processingdevice 314.

The storage device 313, for example, is configured of a hard disk drive,a semiconductor memory, or the like, and stores the training datasupplied from the training data generating processing device 314.

The training data generating processing device 314 performs theprocessing of automatically generating the training data to be used forthe machine learning on the basis of the multispectral image suppliedfrom the multispectral camera 311 and the color image supplied from theRGB camera 312.

In general, spectrum information representing specific spectralcharacteristics with respect to each type of the object, which is asubject, can be extracted from the multispectral image acquired byimaging the object with the light dispersed into a plurality ofwavelength bands. Accordingly, the object photographed in themultispectral image can be recognized with a high accuracy, on the basisof the spectral characteristics with respect to each of the objects. Onthe other hand, the color image is obtained by only imaging the objectwith light in three wavelength bands (for example, R, G, and B), andthus, in order to recognize the object photographed in the color imagewith a high accuracy, it is necessary to perform the machine learningusing a large quantity of training data in advance. For this reason, itis necessary to automatically generate the training data to be used inthe machine learning for recognizing the object of the color image, forexample, a color image to which a name representing the type of theobject is added.

As illustrated in the drawing, the training data generating processingdevice 314 includes a spectrum information retaining unit 321, arecognition processing unit 322, and a labeling unit 323.

The spectrum information retaining unit 321 retains the spectruminformation which is obtained in advance from the multispectral image inwhich various objects are photographed and represents specific spectralcharacteristics with respect to each type of the object, in associationwith the name representing each type of the object.

The recognition processing unit 322 extracts spectral characteristics ina region where the object, which is the target generating the trainingdata in the multispectral image supplied from the multispectral camera311, is photographed, and thus, acquires the spectrum informationrepresenting the spectral characteristics of the object. Further, therecognition processing unit 322 obtains a similarity ratio with respectto a plurality of spectrum information items retained in the spectruminformation retaining unit 321, with respect to the spectrum informationobtained from the multispectral image.

Then, the recognition processing unit 322 supplies a name associatedwith spectrum information having the highest similarity ratio as arecognition result of the object, which is the target generating thetraining data, from the plurality of spectrum information items retainedin the spectrum information retaining unit 321, to the labeling unit323. Furthermore, in a case where the highest similarity ratio obtainedhere is less than or equal to a predetermined defined value (a thresholdvalue which can be determined as the same type), the recognitionprocessing unit 322 sets the effect that the object photographed in themultispectral image is not capable of being recognized (unrecognizable)as the recognition result.

The labeling unit 323 performs labeling processing with respect to thecolor image supplied from the RGB camera 312, that is, processing ofadding the recognition result supplied from the recognition processingunit 322 to the color image supplied from the RGB camera 312. Then, thelabeling unit 323 supplies the color image to which the recognitionresult is added, to the storage device 313 for storage as the trainingdata to be used for the machine learning for performing the objectrecognition using the color image.

The training data generating system 301 is configured as describedabove, and is capable of generating the color image to which therecognition result of the object photographed in the multispectral imageis added, as the training data to be used for the machine learning forperforming the object recognition using the color image. Accordingly,various objects are imaged by the multispectral camera 311 and the RGBcamera 312, by using the training data generating system 301, and thus,the training data with respect to such objects can be automaticallygenerated.

Accordingly, in the related art, for example, a huge labor of visuallydetermining the type of the object, which is the target generating thetraining data and manually performing the labeling is necessary, but thelabor can be reduced due to the automation of the training datagenerating system 301. That is, a large quantity of training data can berapidly generated by the training data generating system 301.

Then, it is possible to perform the machine learning for performing theobject recognition using the color image by using a large quantity oftraining data which is automatically generated by the training datagenerating system 301.

FIG. 24 illustrates a configuration example of the object recognitionsystem 302 performing machine learning by using the training datagenerated in the training data generating system 301, and performing theobject recognition by using a learning result thereof.

As illustrated in FIG. 24 , the object recognition system 302 includes astorage device 331, a learning tool 332, an RGB camera 333, and anoutput device 334. Furthermore, when the learning is performed in theobject recognition system 302, the object recognition system 302 mayinclude at least the storage device 331 and the learning tool 332. Inaddition, when the learning result is used in the object recognitionsystem 302, the object recognition system 302 may include at least thelearning tool 332, the RGB camera 333, and the output device 334.

The storage device 331 stores a large quantity of training data which isautomatically generated in the training data generating system 301 ofFIG. 23 , that is, the color image to which the recognition result withrespect to various objects (the name representing the type of theobject) is added.

The learning tool 332 sequentially reads out a large quantity oftraining data which is stored in the storage device 331 at the time ofperforming the learning, and for example, performs the learning ofextracting the common characteristics for the objects with respect to aplurality of images in which the same type of the object isphotographed. Then, when the object recognition is performed by usingthe learning result, the learning tool 332 compares the characteristicsof the object photographed in the color image which is imaged by the RGBcamera 333 with the characteristics which are subjected to the learningin advance. Accordingly, the learning tool 332 recognizes the type ofthe object photographed in the color image which is imaged by the RGBcamera 333, and supplies the recognition result (the name representingthe type of the object) to the output device 334.

The RGB camera 333 is an imaging device which is capable of imaging ageneral color image, and supplies the color image obtained by imagingthe object, which is the target to be subjected to the objectrecognition, to the learning tool 332.

The output device 334, for example, is configured of a display such as aliquid crystal panel or an organic electro luminescence (EL) panel, anddisplays the recognition result supplied from the learning tool 332 bysuperimposing the recognition result on the color image which is imagedby the RGB camera 333. Furthermore, in a case where the output device334 is configured of a speaker, and a synthetic audio expressing therecognition result may be output.

Thus, the object recognition system 302 performs the machine learning byusing the training data which is generated in the training datagenerating system 301, and is capable of performing the objectrecognition with a higher accuracy by using the learning result.

FIG. 25 is a flowchart illustrating training data generating processingof the training data generating system 301.

For example, in a state where the multispectral camera 311 and the RGBcamera 312 are directed towards the object, which is the targetgenerating the training data, in the case of performing an operation ofinstructing the training data of the object to be generated, theprocessing is started. In step S11, the multispectral camera 311supplies the multispectral image in which the object as the target isphotographed, to the recognition processing unit 322, and the RGB camera312 supplies the color image in which the object as the target isphotographed, to the labeling unit 323.

In step S12, the recognition processing unit 322 extracts the spectralcharacteristics from the multispectral image which is supplied from themultispectral camera 311 in step S11, and acquires the spectruminformation representing the spectral characteristics of the object,which is the target generating the training data.

In step S13, the recognition processing unit 322 obtains the similarityratio with respect to the plurality of spectrum information items whichare retained in the spectrum information retaining unit 321, withrespect to the spectrum information acquired in step S12. Then, therecognition processing unit 322 obtains the name associated with thespectrum information in which the highest similarity ratio is obtained,as the recognition result of the object, which is the target generatingthe training data, and supplies the name to the labeling unit 323.

In step S14, the labeling unit 323 labels the recognition resultsupplied from the recognition processing unit 322 in step S13 withrespect to the color image supplied from the RGB camera 312 in step S11,and generates the training data of the object as the target. Then, thelabeling unit 323 supplies the generated training data to the storagedevice 313.

In step S15, the storage device 313 stores the training data suppliedfrom the labeling unit 323 in step S14, and then, ends the training datagenerating processing. After that, for example, in the case ofperforming an operation of instructing the training data to be generatedby using the next object as the target, hereinafter, similar processingis repeated.

As described above, in the training data generating system 301, it ispossible to automate the labeling processing with respect to the object,which is the target generating the training data, and to easily generatea large quantity of training data.

<Second Usage Example>

FIG. 26 illustrates a configuration example of an evaluation indexpresenting system performing processing of presenting a suitableevaluation index with respect to the object as the target, as a secondusage example of the processing which is performed by using themultispectral image.

As illustrated in FIG. 26 , the evaluation index presenting system 303includes a multispectral camera 311, an RGB camera 312, an output device315, and an evaluation index acquisition processing device 316.Furthermore, the multispectral camera 311 and the RGB camera 312 have asimilar configuration to that of the training data generating system 301of FIG. 23 , and the detailed description will be omitted.

The output device 315, for example, is configured of a display such as aliquid crystal panel or an organic EL panel, and displays the evaluationindex supplied from the evaluation index acquisition processing device316 by superimposing the evaluation index on the color image which isimaged by the RGB camera 312. Furthermore, in a case where the outputdevice 315 is configured of a speaker, a synthetic audio expressing theevaluation index may be output.

The evaluation index acquisition processing device 316 recognizes theobject as the target, on the basis of the multispectral image suppliedfrom the multispectral camera 311 and the color image supplied from theRGB camera 312, and performs processing of acquiring the evaluationindex quantitatively evaluating the object.

As illustrated in the drawing, the evaluation index acquisitionprocessing device 316 includes a spectrum information retaining unit321, a recognition processing unit 322, a labeling unit 323, and anevaluation index calculating unit 324. Furthermore, the spectruminformation retaining unit 321, the recognition processing unit 322, andthe labeling unit 323 have a similar configuration to that of thetraining data generating processing device 314 of FIG. 23 , and thedetailed description will be omitted.

A recognition result of the recognition processing unit 322 is suppliedto the evaluation index calculating unit 324, along with themultispectral image imaged by the multispectral camera 311. Then, theevaluation index calculating unit 324 automatically selects theevaluation index suitable for the type of the object according to therecognition result supplied from the recognition processing unit 322,that is, the name representing the type of the object, which is thetarget calculating the evaluation index. For example, the evaluationindex calculating unit 324 retains the type of the object and theoptimal evaluation index with respect to the type of the object inassociation with each other. Then, as described below with reference toFIGS. 27A and 27B, in a case where a tomato is the target, a sugarcontent is selected as the evaluation index, and in a case where acabbage is the target, freshness is selected as the evaluation index.

Further, the evaluation index calculating unit 324 automatically selectsan index calculation formula and a coefficient necessary for calculatingthe evaluation index suitable for the type of the object as the target,calculates the evaluation index on the basis of the multispectral image,and supplies the evaluation index to the labeling unit 323.

Accordingly, the labeling unit 323 performs the labeling processing withrespect to the color image supplied from the RGB camera 312, that is,processing of adding the recognition result supplied from therecognition processing unit 322 and the evaluation index supplied fromthe evaluation index calculating unit 324 to the color image suppliedfrom the RGB camera 312. Then, the labeling unit 323 supplies the colorimage to which the recognition result and the evaluation index areadded, to the output device 315 for display as the evaluation result.

The evaluation index presenting system 303 is configured as describedabove, and thus, it is possible to display the color image to which therecognition result of the object photographed in the multispectral imageand the evaluation index are added, on the display of the output device315. Accordingly, the evaluation index presenting system 303 is capableof automatically presenting the evaluation index suitable for the objectby only imaging a desired object with the multispectral camera 311 andthe RGB camera 312.

Accordingly, in the related art, for example, an evaluation indexsuitable for each food is different, and thus, in a case where it isnecessary for the user to activate dedicated software with respect toeach of the foods or to set an operation mode, a coefficient, or thelike, such a labor can be reduced by the automation of the evaluationindex presenting system 303. That is, the user may only activate thesoftware presenting the evaluation index regardless of the object as thetarget, and thus, more excellent user experience can be provided by theevaluation index presenting system 303.

FIGS. 27A and 27B illustrate usage examples of an information processingterminal on which the evaluation index presenting system 303 is mounted.

For example, as illustrated in FIG. 27A, in a tomato selling space, itis automatically recognized that the target presenting the evaluationindex is a tomato, by only imaging the tomato with the multispectralcamera 311 and the RGB camera 312 of the information processingterminal. Then, a sugar content is selected as the evaluation indexapplied to the tomato, and an evaluation result of “SUGAR CONTENT OFTHIS TOMATO IS 14.0” with respect to the tomato as the target ispresented on the display, which is the output device 315 of theinformation processing terminal, by being superimposed on the colorimage imaged by the RGB camera 312. Furthermore, for example, a messageof “DURING RECOGNITION OF TYPE OF FOOD” representing that it is in themiddle of the processing is displayed during object recognitionprocessing or the acquisition of the evaluation index.

Similarly, for example, as illustrated in FIG. 27B, in a cabbage sellingspace, it is automatically recognized that the target presenting theevaluation index is a cabbage by only imaging the cabbage with themultispectral camera 311 and the RGB camera 312 of the informationprocessing terminal. Then, freshness is selected as the evaluation indexapplied to the cabbage, and an evaluation result of “FRESHNESS OFCABBAGE IS 70%” with respect to the cabbage as the target is presentedon the display, which is the output device 315 of the informationprocessing terminal by being superimposed on the color image imaged bythe RGB camera 312. Furthermore, for example, a message of “DURINGRECOGNITION OF TYPE OF FOOD” representing that it is in the middle ofthe processing is displayed during the object recognition processing andthe acquisition of the evaluation index.

In addition, the evaluation index presenting system 303, for example, iscapable of obtaining the tastiness of the food as described below withreference to FIG. 32 , the moisture of the fruit as described below withreference to FIG. 33 , and the like as the evaluation index.

FIG. 28 is a flowchart illustrating evaluation index presentingprocessing of the evaluation index presenting system 303.

For example, in the case of performing an operation of instructing theevaluation index of the object to be presented in a state where themultispectral camera 311 and the RGB camera 312 are directed towards theobject, which is the target presenting the evaluation index, theprocessing is started. Then, in steps S21 to S23, similar processing tothat in steps S11 to S13 of FIG. 25 is performed.

In step S24, the evaluation index calculating unit 324 automaticallyselects the optimal evaluation index with respect to the type of theobject as the target, according to a recognition result supplied fromthe recognition processing unit 322 in step S23.

In step S25, the evaluation index calculating unit 324 automaticallyselects an index calculation formula and a coefficient necessary forcalculating the evaluation index selected in step S24, calculates theevaluation index, and supplies the evaluation index to the labeling unit323.

In step S26, the labeling unit 323 labels the recognition resultsupplied from the recognition processing unit 322 in step S23 and theevaluation index supplied from the evaluation index calculating unit 324in step S25, on the color image supplied from the RGB camera 312 in stepS21, and generates the evaluation result of the object as the target.Then, the labeling unit 323 supplies the generated evaluation result tothe output device 315.

In step S27, the output device 315 outputs the evaluation resultsupplied from the labeling unit 323 in step S26, and then, theevaluation index presenting processing is ended. After that, forexample, in the case of performing an operation of instructing theevaluation index to be presented by using the next object as the target,hereinafter, similar processing is repeated.

As described above, in the evaluation index presenting system 303, it ispossible to automatically present the evaluation result applied to theobject by only imaging the object, which is the target presenting theevaluation index, and for example, it is possible to provide the userwith more excellent experience.

FIG. 29 is a block diagram illustrating a configuration example ofhardware of a computer which executes a set of processings describedabove (the training data generating processing of FIG. 25 and theevaluation index presenting processing of FIG. 28 ) by a program.

In the computer, a central processing unit (CPU) 401, a read only memory(ROM) 402, a random access memory (RAM) 403, and an electronicallyerasable and programmable read only memory (EEPROM) 404 are connected toeach other through a bus 405. An input/output interface 406 is furtherconnected to the bus 405, and the input/output interface 406 isconnected to the outside (for example, the multispectral camera 311, theRGB camera 312, or the like).

In the computer configured as described above, the CPU 401, for example,loads a program stored in the ROM 402 and the EEPROM 404 on the RAM 403through the bus 405, and executes the program, and thus, performs theset of processings described above. In addition, the program executed bythe computer (the CPU 101) is written in advance in the ROM 402, and canbe installed in the EEPROM 404 from the outside through the input/outputinterface 406 or can be updated.

Furthermore, the training data generating system 301 and the evaluationindex presenting system 303 include two cameras of the multispectralcamera 311 and the RGB camera 312, and for example, may include onecamera which is capable of detecting a multiwavelength.

In addition, the training data generating system 301 and the evaluationindex presenting system 303 may be incorporated in the imaging device 10including the imaging element 12 of FIG. 1 , and for example, thetraining data, the evaluation result, or the like can be output from theimaging device 10.

Furthermore, the training data generating system 301 and the evaluationindex presenting system 303 are not limited to recognizing the object byonly using the multispectral camera 311 as described above, and mayrecognize the object by using other types of sensors. For example, abrightness sensor acquiring brightness information, a time-of-flight(ToF) sensor acquiring distance information, an ambient sensor acquiringvarious environmental information items, an infrared ray sensoracquiring infrared ray information, a thermal sensor acquiringtemperature information, and the like can be used, and an imageconfigured of the information acquired by such sensors can be used forrecognizing the object.

Further, the labeling processing which is automated by using themultispectral image output from the imaging element 12 of FIG. 1 is notlimited to the recognition result or the evaluation result of theobject, and the labeling of various information items can be performed.

3. Modification Example

Hereinafter, a modification example of the embodiments of the presenttechnology described above will be described.

For example, the number of types of the film thicknesses of theconductor thin film may be set to be greater than or equal to 3,according to the hole pitch (the transmission band).

In addition, in the plasmon filter having the dot array structure, thefilm thickness of the conductor thin film (the dot) may be changedaccording to the dot pitch (the absorption band).

Specifically, as illustrated in FIG. 13 , as the dot pitch narrows andthe absorption band is shifted to a short wavelength, a peak width and ahalf width of the absorption band basically narrow, but an absorptionrate (a negative peak value of the absorption band) decreases. Incontrast, as the dot pitch widens and the absorption band is shifted toa long wavelength, the absorption rate (the negative peak value of theabsorption band) is basically improved, but the peak width and the halfwidth of the absorption band widen.

In addition, as the conductor thin film configuring the dot becomesthin, the absorption rate basically decreases, but the peak width andthe half width of the absorption band narrow. In contrast, as theconductor thin film configuring the dot becomes thick, the peak widthand the half width of the absorption band basically widen, but theabsorption rate is improved.

Accordingly, for example, it is desirable that as the dot pitch of theplasmon filter narrows and the absorption band is shifted to the shortwavelength, the conductor thin film becomes thick and the absorptionrate increases, even though the peak width and the half width of theabsorption band slightly widen. In contrast, it is desirable that as thedot pitch of the plasmon filter widens and the absorption band isshifted to the long wavelength, the conductor thin film becomes thin andthe peak width and the half width of the transmission band narrow, eventhough the absorption rate slightly decreases.

Further, for example, in the plasmon filter of the same transmissionband (the same hole pitch) or the same absorption band (the same dotpitch), the film thickness of the conductor thin film may be changed foreach pixel. Accordingly, it is possible to provide pixels of which thetransmission bands or the absorption bands are identical to each other,but the sensitivities or the absorption rates are different from eachother. Accordingly, for example, it is possible to improve a detectionaccuracy of narrow band light in a part of the pixels.

In addition, the present technology is not limited only to the back-sideillumination type CMOS image sensor described above, but can be appliedto other imaging elements using the plasmon filter. For example, thepresent technology can be applied to a surface irradiation type CMOSimage sensor, a charge coupled device (CCD) image sensor, an imagesensor having a photoconductor structure in which an organicphotoelectric conversion film, a quantum dot structure, or the like isembedded, and the like.

In addition, the present technology, for example, can be applied to alaminated solid imaging device illustrated in FIGS. 30A to 30C.

FIG. 30A illustrates a schematic configuration example of anon-laminated solid imaging device. As illustrated in FIG. 30A, a solidimaging device 1010 includes one die (a semiconductor substrate) 1011. Apixel region 1012 in which the pixels are arranged in the shape of anarray, a control circuit 1013 performing various controls other than thedriving of the pixel, and logic circuit 1014 for signal processing aremounted on the die 1011.

FIGS. 30B and 30C illustrate schematic configuration examples of alaminated solid imaging device. As illustrated in FIGS. 30B and 30C, twodies of a sensor die 1021 and a logic die 1022 are laminated on a solidimaging device 1020, are electrically connected to each other, and areconfigured as one semiconductor chip.

In FIG. 30B, the pixel region 1012 and the control circuit 1013 aremounted on the sensor die 1021, and the logic circuit 1014 including asignal processing circuit which performs the signal processing ismounted on the logic die 1022.

In FIG. 30C, the pixel region 1012 is mounted on the sensor die 1021,and the control circuit 1013 and the logic circuit 1014 are mounted onthe logic die 1024.

Further, the present technology can be applied to a metal thin filmfilter using a metal thin film other than the plasmon filter, and apossibility that the present technology is applied to photonic crystalsusing a semiconductor material is considered as an application example.

4. Application Example

Next, an application example of the present technology will bedescribed.

<Application Example of Present Technology>

For example, as illustrated in FIG. 31 , the present technology can beapplied to various cases of sensing light such as visible light,infrared light, ultraviolet light, and an X ray.

-   -   a device shooting an image provided for viewing, such as a        digital camera or portable device having a camera function    -   a device provided for traffic, such as an in-vehicle sensor        shooting the front side, the rear side, the circumference, the        inside, or the like of the automobile, a monitoring camera        monitoring a running vehicle or a road, and a distance measuring        sensor measuring a distance between vehicles or the like, in        order for a safety operation such as automatic stop, the        recognition of the state of a driver, and the like    -   a device provided for a home electrical appliance, such as a TV,        a refrigerator, and an air conditioner, in order to shoot the        gesture of the user, and to perform a device operation according        to the gesture    -   a device provided for a medical care or a health care, such as        an endoscope or a device performing angiography by receiving        infrared light    -   a device provided for security, such as a monitoring camera for        anti-crime and a camera for personal authentication    -   a device provided for a beauty care, such as a skin measuring        machine shooting the skin and a microscope shooting the scalp    -   a device provided for sport, such as an action camera or a        wearable camera for sport    -   a device provided for agriculture, such as a camera monitoring        the state of the cultivation or the crop

Hereinafter, a more detailed application example will be described.

For example, the transmission band of the narrow band filter NB of eachof the pixels 51 of the imaging device 10 of FIG. 1 is adjusted, andthus, a wavelength band of light which is detected by each of the pixels51 of the imaging device 10 (hereinafter, referred to as a detectionband) can be adjusted. Then, the detection band of each of the pixels 51is suitably set, and thus, the imaging device 10 can be used for variousapplications.

For example, FIG. 32 illustrates an example of a detection band in acase where the tastiness or the freshness of the food is detected.

For example, a peak wavelength of a detection band in the case ofdetecting myoglobin representing a tastiness component of tuna, beef, orthe like is in a range of 580 nm to 630 nm, and a half width is in arange of 30 nm to 50 nm. A peak wavelength of a detection band in thecase of detecting an oleic acid representing the freshness of the tuna,the beef, or the like is 980 nm, and a half width is in a range of 50 nmto 100 nm. A peak wavelength of a detection band in the case ofdetecting chlorophyll representing the freshness of leaf vegetable suchas Brassica rapa is in a range of 650 nm to 700 nm, and a half width isin a range of 50 nm to 100 nm.

FIG. 33 illustrates an example of a detection band in a case where asugar content or the moisture of a fruit is detected.

For example, a peak wavelength of a detection band in the case ofdetecting a flesh light path length representing a sugar content ofRaiden, which is one breed of melon, is 880 nm, and a half width is in arange of 20 nm to 30 nm. A peak wavelength of a detection band in thecase of detecting sucrose representing the sugar content of Raiden is910 nm, and a half width is in a range of 40 nm to 50 nm.

A peak wavelength of a detection band in the case of detecting sucroserepresenting a sugar content of Raiden Red, which is another breed ofmelon, is 915 nm, and a half width is in a range of 40 nm to 50 nm. Apeak wavelength of a detection band in the case of detecting moisturerepresenting the sugar content of Raiden Red is 955 nm, and a half widthis in a range of 20 nm to 30 nm.

A peak wavelength of a detection band in the case of detecting sucroserepresenting a sugar content of an apple is 912 nm, and a half width isin a range of 40 nm to 50 nm. A peak wavelength of a detection band inthe case of detecting water representing the moisture of a mandarinorange is 844 nm, and a half width is 30 nm. A peak wavelength of adetection band in the case of detecting sucrose representing a sugarcontent of the mandarin orange is 914 nm, and a half width is in a rangeof 40 nm to 50 nm.

FIG. 34 illustrates an example of a detection band in a case whereplastics are sorted.

For example, a peak wavelength of a detection band in the case ofdetecting poly ethylene terephthalate (PET) is 1669 nm, and a half widthis in a range of 30 nm to 50 nm. A peak wavelength of a detection bandin the case of detecting poly styrene (PS) is 1688 nm, and a half widthis in a range of 30 nm to 50 nm. A peak wavelength of a detection bandin the case of detecting poly ethylene (PE) is 1735 nm, and a half widthis in a range of 30 nm to 50 nm. A peak wavelength of a detection bandin the case of detecting poly vinyl chloride (PVC) is in a range of 1716nm to 1726 nm, and a half width is in a range of 30 nm to 50 nm. A peakwavelength of a detection band in the case of detecting polypropylene(PP) is in a range of 1716 nm to 1735 nm, and a half width is in a rangeof 30 nm to 50 nm.

In addition, for example, the present technology can be applied tofreshness management of plucked flower.

Further, for example, the present technology can be applied to aninspection of foreign substances which are mixed into the food. Forexample, the present technology can be applied to the detection of theforeign substances, such as a shell, a hull, a stone, a leaf, a branch,and a wood chip, which are mixed into nuts, such as an almond, ablueberry, and a walnut, or fruits. In addition, for example, thepresent technology can be applied to the detection of the foreignsubstances such as plastic pieces mixed into processed food, beverage,or the like.

Further, for example, the present technology can be applied to thedetection of a normalized difference vegetation index (NDVI), which isan index of vegetation.

In addition, for example, the present technology can be applied to thedetection of a human body on the basis of any one or both of a spectralshape in the vicinity of a wavelength of 580 nm, derived from Hemoglobinof the human skin and a spectral shape in the vicinity of a wavelengthof 960 nm, derived from a melanin dye contained in the human skin.

Further, for example, the present technology can be applied tobiological detection (biological authentication), fabricationprevention, monitoring, and the like of a user interface and a sign.

<Application Example of Endoscopic Surgery System>

In addition, for example, a technology according to an embodiment of thepresent disclosure (the present technology) may be applied to anendoscopic surgery system.

FIG. 35 is a diagram illustrating an example of a schematicconfiguration of the endoscopic surgery system to which the technologyaccording to an embodiment of the present disclosure (the presenttechnology) is applied.

FIG. 35 illustrates an aspect in which an operator (a medical doctor)11131 performs a surgery with respect to a patient 11132 on a patientbed 11133 by using an endoscopic surgery system 11000. As illustrated inthe drawing, the endoscopic surgery system 11000 is configured of anendoscope 11100, other surgical tools 11110 such as a pneumoperitoneumtube 11111 or an energy treatment tool 11112, a support arm device 11120supporting the endoscope 11100, and a cart 11200 on which variousdevices for the surgery under the endoscope are mounted.

The endoscope 11100 is configured of a lens barrel 11101 in which aregion having a predetermined length from a tip end is inserted into abody cavity of the patient 11132, and a camera head 11102 connected to abase end of the lens barrel 11101. In the illustrated example, theendoscope 11100 configured as a so-called rigid scope including a rigidlens barrel 11101 is illustrated, and the endoscope 11100 may beconfigured as a so-called flexible scope including a flexible lensbarrel.

An opening portion embedded with an objective lens is disposed on thetip end of the lens barrel 11101. A light source device 11203 isconnected to the endoscope 11100, and light generated by the lightsource device 11203 is guided to the tip end of the lens barrel by alight guide extending in the lens barrel 11101, and is emitted towardsan observation target in the body cavity of the patient 11132 throughthe objective lens. Furthermore, the endoscope 11100 may be a directview mirror, or may be a perspective view mirror or a side view mirror.

An optical system and an imaging element are disposed on the camera head11102, and reflection light from the observation target (observationlight) is condensed on the imaging element by the optical system. Theobservation light is subjected to photoelectric conversion by theimaging element, and thus, an electric signal corresponding to theobservation light, that is, an image signal corresponding to theobservation image is generated. The image signal is transmitted to acamera control unit (CCU) 11201 as RAW data.

The CCU 11201 is configured of a central processing unit (CPU), agraphics processing unit (GPU), or the like, and integrally controls theoperations of the endoscope 11100 and a display device 11202. Further,the CCU 11201 receives the image signal from the camera head 11102, andperforms various image processings for displaying an image based on theimage signal with respect to the image signal, such as developingprocessing (demosaic processing).

The display device 11202 displays the image based on the image signal,which is subjected to the image processing by the CCU 11201, accordingto the control from the CCU 11201.

The light source device 11203, for example, is configured of a lightsource such as a light emitting diode (LED), and supplies irradiationlight at the time of shooting a surgical site or the like to theendoscope 11100.

An input device 11204 is an input interface with respect to theendoscopic surgery system 11000. It is possible for the user to performvarious information inputs or instruction inputs with respect to theendoscopic surgery system 11000 through the input device 11204. Forexample, the user inputs an instruction or the like to the effect ofchanging imaging conditions of the endoscope 11100 (the type ofirradiation light, a magnification, a focal point distance, and thelike).

A treatment tool control device 11205 controls the drive of the energytreatment tool 11112, such as the cauterization of tissues, and theincision or the sealing of a blood vessel. A pneumoperitoneum device11206 feeds gas in the body cavity through the pneumoperitoneum tube11111, in order to inflate the body cavity of the patient 11132 toensure a visual field of the endoscope 11100 and an operation space ofthe operator. A recorder 11207 is a device which is capable of recordingvarious information items relevant to the surgery. A printer 11208 is adevice which is capable of printing various information items relevantto the surgery in various formats such as a text, an image, or a graph.

Furthermore, the light source device 11203 supplying the irradiationlight at the time of shooting the surgical site to the endoscope 11100,for example, can be configured of a white light source which isconfigured of an LED, a laser light source, or a combination thereof. Ina case where the white light source is configured of a combination ofRGB laser light sources, an output intensity and an output timing ofeach color (each wavelength) can be controlled with a high accuracy, andthus, a white balance of the imaged image can be adjusted in the lightsource device 11203. In addition, in this case, the RGB laser lightsource irradiates the observation target with each laser light ray intime division, and controls the driving of the imaging element of thecamera head 11102 in synchronization with the irradiation timing, andthus, it is also possible to image an image corresponding to each of RGBin time division. According to the method described above, it ispossible to obtain a color image even in a case where the color filteris not disposed in the imaging element.

In addition, the light source device 11203 may control the driving suchthat the light intensity to be output is changed for each predeterminedtime. The driving of the imaging element of the camera head 11102 iscontrolled in synchronization with a timing at which the light intensityis changed, an image is acquired in time division, and the image issynthesized, and thus, it is possible to generate an image in a highdynamic range without having so-called black defects and overexposure.

In addition, the light source device 11203 may be configured to becapable of supplying light in a predetermined wavelength bandcorresponding to special light observation. In the special lightobservation, for example, light in a narrow band, compared to theirradiation light (that is, white light) at the time of normalobservation, is emitted by using wavelength dependency of lightabsorption in the body tissues, and thus, so-called narrow band lightobservation (narrow band imaging) shooting a predetermined tissue of theblood vessel or the like on a surface layer of a mucous membrane with ahigh contrast is performed. Alternatively, in the special lightobservation, fluorescent light observation may be performed in which animage is obtained by fluorescent light generated by emitting excitationlight. In the fluorescent light observation, the body tissues areirradiated with the excitation light, and thus, the fluorescent lightfrom the body tissues can be observed (self-fluorescent lightobservation), or a reagent such as indocyanine green (ICG) is locallyinjected into the body tissues, and the body tissues are irradiated withexcitation light corresponding to the wavelength of the fluorescentlight of the reagent, and thus, a fluorescent image can be obtained. Thelight source device 11203 can be configured to be capable of supplyingthe narrow band light and/or the excitation light corresponding to thespecial light observation.

FIG. 36 is a block diagram illustrating an example of functionalconfigurations of the camera head 11102 and the CCU 11201 illustrated inFIG. 35 .

The camera head 11102 includes a lens unit 11401, an imaging unit 11402,a driving unit 11403, a communication unit 11404, and a camera headcontrol unit 11405. The CCU 11201 includes a communication unit 11411,an image processing unit 11412, and a control unit 11413. The camerahead 11102 and the CCU 11201 are connected to each other to communicatewith each other by a transmission cable 11400.

The lens unit 11401 is an optical system which is disposed in aconnection portion with respect to the lens barrel 11101. The capturedobservation light from the tip end of the lens barrel 11101 is guided tothe camera head 11102, and is incident on the lens unit 11401. The lensunit 11401 is configured of a combination of a plurality of lensesincluding a zoom lens and a focus lens.

The imaging element configuring the imaging unit 11402 may be oneimaging element (a so-called single-plate type imaging element), or maybe a plurality of imaging elements (a so-called multi-plate type imagingelement). In a case where the imaging unit 11402 is configured of themulti-plate type imaging element, for example, image signalscorresponding to each of RGB are generated by each of the imagingelements, and are synthesized, and thus, a color image may be obtained.Alternatively, the imaging unit 11402 may be configured to include apair of imaging elements for acquiring image signals for a right eye anda left eye, which correspond to three-dimensional (3D) display. Byperforming the 3D display, it is possible for the operator 11131 to moreaccurately grasp the depth of the body tissues in the surgical site.Furthermore, in a case where the imaging unit 11402 is configured of themulti-plate type imaging element, a plurality of lens units 11401 canalso be disposed corresponding to each of the imaging elements.

In addition, the imaging unit 11402 may not be necessarily disposed onthe camera head 11102. For example, the imaging unit 11402 may disposedin the lens barrel 11101 immediately behind the objective lens.

The driving unit 11403 is configured of an actuator, and moves the zoomlens and the focus lens of the lens unit 11401 along an optical axis bya predetermined distance, according to the control from the camera headcontrol unit 11405. Accordingly, the magnification and the focal pointof the imaged image obtained by the imaging unit 11402 can be suitablyadjusted.

The communication unit 11404 is configured of a communication device fortransmitting and receiving various information items with respect to theCCU 11201. The communication unit 11404 transmits the image signalobtained from the imaging unit 11402 to the CCU 11201 through thetransmission cable 11400, as RAW data.

In addition, the communication unit 11404 receives a control signal forcontrolling the driving the camera head 11102 from the CCU 11201, andsupplies the control signal to the camera head control unit 11405. Thecontrol signal, for example, includes information relevant to theimaging conditions, such as information to the effect of designating aframe rate of the imaged image, information to the effect of designatingan exposure value at the time of imaging, and/or information to theeffect of designating the magnification and the focal point of theimaged image.

Furthermore, the imaging conditions such as the frame rate or theexposure value, the magnification, and the focal point, described above,may be suitably designated by the user, or may be automatically set bythe control unit 11413 of the CCU 11201 on the basis of the acquiredimage signal. In the latter case, a so-called auto exposure (AE)function, an auto focus (AF) function, and an auto white balance (AWB)function are mounted on the endoscope 11100.

The camera head control unit 11405 controls the driving of the camerahead 11102 on the basis of the control signal from the CCU 11201, whichis received through the communication unit 11404.

The communication unit 11411 is configured of a communication device fortransmitting and receiving various information items with respect to thecamera head 11102. The communication unit 11411 receives the imagesignal transmitted through the transmission cable 11400 from the camerahead 11102.

In addition, the communication unit 11411 transmits the control signalfor controlling the driving of the camera head 11102 to the camera head11102. The image signal or the control signal can be transmitted bytelecommunication, light communication, or the like.

The image processing unit 11412 performs various image processings withrespect to the image signal, which is the RAW data transmitted from thecamera head 11102.

The control unit 11413 performs various controls relevant to the imagingof the surgical site or the like using the endoscope 11100 and thedisplay of the imaged image obtained by imaging the surgical site or thelike. For example, the control unit 11413 generates the control signalfor controlling the driving of the camera head 11102.

In addition, the control unit 11413 displays the imaged image, on whichthe surgical site or the like is reflected, on the display device 11202,on the basis of the image signal which is subjected to the imageprocessing by the image processing unit 11412. At this time, the controlunit 11413 may recognize various objects in the imaged image by usingvarious image recognition technologies. For example, the control unit11413 detects the shape, the color, or the like of the edge of theobject which is included in the imaged image, and thus, is capable ofrecognizing a surgical tool such as forceps, a specific organic site,bleed, mist at the time of using the energy treatment tool 11112, or thelike. The control unit 11413 may display various surgery assistanceinformation items by superimpose the information on the image of thesurgical site, by using the recognition result, at the time ofdisplaying the imaged image on the display device 11202. The surgeryassistance information is displayed by being superimposed, and ispresented to the operator 11131, and thus, it is possible to reduce aload on the operator 11131, and it is possible for the operator 11131 toreliably perform the surgery.

The transmission cable 11400 connecting the camera head 11102 and theCCU 11201 to each other is an electric signal cable corresponding to thecommunication of the electric signal, an optical fiber corresponding tothe light communication, or a composite cable thereof.

Here, in the illustrated example, the communication is performed in awired manner by using the transmission cable 11400, and thecommunication between the camera head 11102 and the CCU 11201 may beperformed in a wireless manner.

As described above, an example of the endoscopic surgery system whichcan be obtained by applying the technology according to an embodiment ofthe present disclosure thereto has been described. In the configurationsdescribed above, the technology according to an embodiment of thepresent disclosure, for example, can be obtained by being applied to thecamera head 11102 or the imaging unit 11402 of the camera head 11102.Specifically, for example, the imaging element 12 of FIG. 1 can beapplied to the imaging unit 11402. It is possible to obtain a morespecific and high accurate surgical site image by applying thetechnology according to an embodiment of the present disclosure to theimaging unit 11402, and thus, it is possible for the operator toreliably confirm the surgical site.

Furthermore, here, the endoscopic surgery system has been described asan example, but the technology according to an embodiment of the presentdisclosure, for example, may be applied to a microscope surgery systemor the like in addition to the endoscopic surgery system.

<Application Example to Movable Body>

In addition, for example, the technology according to an embodiment ofthe present disclosure may be realized as a device mounted on any typeof movable body such as an automobile, an electric automobile, a hybridelectric automobile, a motorcycle, a bicycle, a personal mobility, anairplane, a drone, a ship, and a robot.

FIG. 37 is a block diagram illustrating a schematic configurationexample of a vehicle control system, which is an example of a movablebody control system obtained by applying the technology according to anembodiment of the present disclosure thereto.

A vehicle control system 12000 includes a plurality of electroniccontrol units connected to each other through a communication network12001. In the example illustrated in FIG. 37 , the vehicle controlsystem 12000 includes a driving system control unit 12010, a body systemcontrol unit 12020, an outdoor information detection unit 12030, anin-vehicle information detection unit 12040, and an integral controlunit 12050. In addition, a microcomputer 12051, an audio image outputunit 12052, and an in-vehicle network interface (I/F) 12053 areillustrated as a functional configuration of the integral control unit12050.

The driving system control unit 12010 controls an operation of a devicerelevant to a driving system of the vehicle according to variousprograms. For example, the driving system control unit 12010 functionsas a control device of a driving force generating device for generatinga driving force of a vehicle, such as an internal-combustion engine or adriving motor, a driving force transfer mechanism for transferring thedriving force to a wheel, a steering mechanism adjusting a rudder angleof the vehicle, a braking device generating a braking force of thevehicle, and the like.

The body system control unit 12020 controls the operations of variousdevices mounted on a vehicle body according to various programs. Forexample, the body system control unit 12020 functions as a controldevice of a keyless entry system, a smart key system, an electric windowdevice, and various lamps such as a head lamp, a back lamp, a brakelamp, a winker lamp, or a fog lamp. In this case, an electric wavetransmitted from a portable machine instead of a key or signals ofvarious switches can be input into the body system control unit 12020.The body system control unit 12020 receives the input of the electricwave or the signal, and controls the door lock device, the electricwindow device, the lamp, and the like of the vehicle.

The outdoor information detection unit 12030 detects the outsideinformation of the vehicle on which the vehicle control system 12000 ismounted. For example, an imaging unit 12031 is connected to the outdoorinformation detection unit 12030. The outdoor information detection unit12030 images the outdoor image by the imaging unit 12031, and receivesthe imaged image. The outdoor information detection unit 12030 mayperform object detection processing or distance detection processing ofa person, a car, an obstacle, a sign, characters on a road surface, orthe like, on the basis of the received image.

The imaging unit 12031 is an optical sensor which receives light andoutputs an electric signal according to the amount of the receivedlight. The imaging unit 12031 is capable of outputting the electricsignal as an image, and is capable of outputting the electric signal asdistance measuring information. In addition, the light received by theimaging unit 12031 may be visible light, or may be non-visible lightsuch as an infrared ray.

The in-vehicle information detection unit 12040 detects in-vehicleinformation. For example, a driver state detecting unit 12041 detectingthe state of the driver is connected to the in-vehicle informationdetection unit 12040. The driver state detecting unit 12041, forexample, includes a camera imaging the driver, and the in-vehicleinformation detection unit 12040 may calculate a fatigue degree or aconcentration degree of the driver, or may determine whether or not thedriver dozes off, on the basis of detection information input from thedriver state detecting unit 12041.

The microcomputer 12051 calculates a control target value of the drivingforce generating device, the steering mechanism, or the braking deviceon the basis of the in-vehicle information and the outdoor information,which are acquired in the outdoor information detection unit 12030 orthe in-vehicle information detection unit 12040, and is capable ofoutputting a control command to the driving system control unit 12010.For example, the microcomputer 12051 is capable of performingcooperative control for realizing the function of an advanced driverassistance system (ADAS) including collision avoidance or impactrelaxation of the vehicle, following running based on an inter-vehicledistance, vehicle speed maintaining running, collision warning of thevehicle, lane departure warning of the vehicle, and the like.

In addition, the microcomputer 12051 controls driving force generatingdevice, the steering mechanism, the braking device, or the like, on thebasis of the information around the vehicle, which is acquired in theoutdoor information detection unit 12030 or the in-vehicle informationdetection unit 12040, and is capable of performing cooperative controlfor automated driving in which the vehicle autonomously runs withoutdepending on the operation of the driver.

In addition, the microcomputer 12051 is capable of outputting thecontrol command to the body system control unit 12020, on the basis ofthe outdoor information, which is acquired in the outdoor informationdetection unit 12030. For example, the microcomputer 12051 controls thehead lamp according to the position of a leading vehicle or an oncomingvehicle, which is detected by the outdoor information detection unit12030, and thus, is capable of performing cooperative control forglare-proof such as switching the high beam with a low beam.

The audio image output unit 12052 transmits at least one output signalof an audio and an image to an output device which is capable ofvisually or auditorily notifying a person on board or the outdoor of thevehicle of the information. In the example of FIG. 37 , an audio speaker12061, a display unit 12062, and an instrument panel 12063 areexemplified as the output device. The display unit 12062, for example,may include at least one of an on-board display and a head-up display.

FIG. 38 is a diagram illustrating an example of a disposition positionof the imaging unit 12031.

In FIG. 38 , the imaging unit 12031 includes imaging units 12101, 12102,12103, 12104, and 12105.

The imaging units 12101, 12102, 12103, 12104, and 12105, for example,are disposed in positions such as a front nose, a side mirror, a rearbumper, a back door of a vehicle 12100, and an upper portion of a frontglass of a vehicle interior. The imaging unit 12101 provided in thefront nose and the imaging unit 12105 provided in the upper portion ofthe front glass of the vehicle interior mainly acquire a front image ofthe vehicle 12100. The imaging units 12102 and 12103 provided in theside mirror mainly acquire a side image of the vehicle 12100. Theimaging unit 12104 provided in the rear bumper or the back door mainlyacquires a rear image of the vehicle 12100. The imaging unit 12105provided in the upper portion of the front glass of the vehicle interioris mainly used for detecting a leading vehicle, a pedestrian, anobstacle, a traffic light, a traffic sign, a traffic lane, or the like.

Furthermore, FIG. 38 illustrates an example of shooting ranges of theimaging units 12101 to 12104. The imaging range 12111 illustrates animaging range of the imaging unit 12101 provided in the front nose,imaging ranges 12112 and 12113 illustrate imaging ranges of the imagingunits 12102 and 12103 respectively provided in the side mirror, and theimaging range 12114 illustrates an imaging range of the imaging unit12104 provided in the rear bumper or the back door. For example, imagedata items imaged in the imaging units 12101 to 12104 are superimposedon each other, and thus, an overhead image is obtained in which thevehicle 12100 is viewed from the upper side.

At least one of the imaging units 12101 to 12104 may have a function ofacquiring the distance information. For example, at least one of theimaging units 12101 to 12104 may be a stereo camera formed of aplurality of imaging elements, or may be an imaging element including apixel for detecting a phase difference.

For example, the microcomputer 12051 obtains a distance to each solidobject in the imaging ranges 12111 to 12114, and a temporal change ofthe distance (a relative speed with respect to the vehicle 12100), onthe basis of the distance information obtained from the imaging units12101 to 12104, and thus, in particular, it is possible to extract thesolid object running at a predetermined speed (for example, greater thanor equal to 0 km/h) in approximately the same direction as that of thevehicle 12100 as the leading vehicle, in the closest solid object on atraveling path of the vehicle 12100. Further, the microcomputer 12051sets the inter-vehicle distance to be ensured in advance immediatelybefore the leading vehicle, and thus, is capable of performing automaticbrake control (also including following stop control), automaticacceleration control (also including following start control), or thelike. Thus, it is possible to perform the cooperative control for theautomated driving in which the vehicle autonomously runs withoutdepending on the operation of the driver.

For example, it is possible for the microcomputer 12051 to extract solidobject data relevant to the solid object by sorting the data into othersolid objects such as a two-wheeled vehicle, an ordinary vehicle, alarge vehicle, a pedestrian, and a telegraph pole, on the basis of thedistance information obtained from the imaging units 12101 to 12104, andto use the data for automatically avoiding the obstacle. For example,the microcomputer 12051 distinguishes the obstacle around the vehicle12100 between an obstacle which is visible to the driver of the vehicle12100 and an obstacle which is not visible. Then, the microcomputer12051 determines collision risk representing a dangerous extent of thecollision with respect to each of the obstacles, and in the case of asituation in which the collision risk is greater than or equal to a setvalue, that is, there is a possibility of the collision, an alarm isoutput to the driver through the audio speaker 12061 or the display unit12062, or forced deceleration and avoidance steering is performedthrough the driving system control unit 12010, and thus, it is possibleto perform driving assistance for avoiding the collision.

At least one of the imaging units 12101 to 12104 may be an infrared raycamera detecting an infrared ray. For example, the microcomputer 12051determines whether or not the pedestrian exists in the imaged images ofthe imaging units 12101 to 12104, and thus, it is possible to recognizethe pedestrian. Such recognition of the pedestrian, for example, isperformed in the order of extracting a characteristic point in theimaged images of the imaging units 12101 to 12104 as the infrared raycamera and the order of determining whether or not there is thepedestrian by performing pattern matching processing with respect to aset of characteristic points representing the outline of the object. Themicrocomputer 12051 determines that the pedestrian exists in the imagedimages of the imaging units 12101 to 12104, and in a case where thepedestrian is recognized, the audio image output unit 12052 controls thedisplay unit 12062 such that a rectangular outline for emphasis isdisplayed by being superimposed on the recognized pedestrian. Inaddition, the audio image output unit 12052 may control the display unit12062 such that an icon or the like representing the pedestrian isdisplayed in a desired position.

As described above, an example of the vehicle control system, which canbe obtained by applying the technology according to an embodiment of thepresent disclosure thereto, has been described. In the configurationsdescribed above, the technology according to an embodiment of thepresent disclosure, for example, can be applied to the imaging unit12031. Specifically, for example, the imaging device 10 of FIG. 1 can beapplied to the imaging unit 12031. By applying the technology accordingto an embodiment of the present disclosure to the imaging unit 12031,for example, it is possible to more specifically acquire the outdoorinformation with a higher accuracy, and to realize improvement or thelike or the safety of the automated driving.

Furthermore, the embodiment of the present technology are not limited tothe embodiments described above, and can be variously changed within arange not departing from the gist of the present technology.

Additionally, the present technology may also be configured as below.

-   -   (1)    -   An imaging system, comprising:        -   a multispectral camera configured to capture a multispectral            image of an object;        -   an RGB camera configured to capture a color image of the            object;    -   at least one storage device configured to store spectrum        information for each of a plurality of labeled objects; and        -   processing circuitry configured to:    -   determine, based on the captured multispectral image, spectrum        information associated with the object; associate, based at        least in part, on the spectrum information associated with the        object and the stored spectrum information for each of the        plurality of objects, a label with the color image of the        object; and    -   store, on the at least one storage device, the color image and        the associated label as training data.    -   (2)    -   The imaging system of claim 1, wherein associating a label with        the color image of the object comprises:        -   determining a similarity measure of the spectrum information            of the object with the stored spectrum information for each            of the plurality of labeled objects; identifying, based on            the determined similarity measures, the labeled object            having a highest similarity measure; and        -   associating a label associated with the labeled object            having the highest similarity measure with the color image            of the object.    -   (3)    -   The imaging system of claim 2, wherein the similarity measure        comprises a similarity ratio.    -   (4)    -   The imaging system of any one of (2) or (3), wherein associating        a label with the color image of the object further comprises:        -   comparing the similarity measure for the labeled object            having a highest similarity measure to a threshold value;            and        -   associating the label associated with the labeled object            having the highest similarity measure with the color image            of the object only when the similarity measure for the            labeled object is greater than the threshold value.    -   (5)    -   The imaging system of any one of (1) to (4), wherein determining        spectrum information associated with the object comprises:        -   identifying a region of the multispectral image that            includes the object, wherein the region comprises a region            smaller than the entire image; and        -   determining the spectrum information associated with the            object based on the region of the multispectral image.    -   (6)    -   The imaging system of any one of (1) to (5), wherein the        multispectral camera and the RGB camera are configured to        capture the multispectral image and the color image,        respectively, simultaneously.    -   (7)    -   The imaging system of any one of (1) to (6), wherein the        processing circuitry is further configured to train an object        classifier using the stored training data to generate a trained        object classifier.    -   (8)    -   The imaging system of (7), wherein the processing circuitry is        further configured to classify an object in a received color        image using the trained object classifier.    -   (9)    -   The imaging system of (8), wherein the processing circuitry is        further configured to:        -   determine based, at least in part, on the classification of            the object in the received color image and spectrum            information associated with the object, an evaluation index            value for a characteristic of the object; and        -   output on a display, an indication of the evaluation index            value.    -   (10)    -   The imaging system of (9), wherein the processing circuity is        further configured to output on the display, the received color        image, and an indication of the classification of the object in        the color image.    -   (11)    -   The imaging system of (9), wherein the indication of the        classification value comprises text and wherein the processing        circuity is further configured to output on the display, the        text superimposed on the received color image.    -   (12)    -   The imaging system of (9), wherein determining the evaluation        index value for a characteristic of the object comprises:        -   selecting based at least on part, on the classification of            the object, an index calculating formula and a coefficient            necessary for calculating the evaluation index value; and        -   determining the evaluation index value using the selected            index calculating formula and coefficient.    -   (13)    -   The imaging system of (9), wherein the processing circuity is        further configured to determine the spectrum information        associated with the object.    -   (14)    -   An object classification system, comprising:        -   at least one storage device configured to store a trained            object classifier; and        -   processing circuitry configured to:            -   classify an object in a received color image using the                trained object classifier;            -   determine based, at least in part, on the classification                of the object in the received color image and spectrum                information associated with the object, an evaluation                index value for a characteristic of the object; and            -   output on a display, an indication of the evaluation                index value.    -   (15)    -   The object classification system of (14), wherein the processing        circuity is further configured to output on the display, the        received color image, and an indication of the classification of        the object in the color image.    -   (16)    -   The object classification system of any one of (14) or (15),        wherein determining the evaluation index value for a        characteristic of the object comprises:        -   selecting based at least on part, on the classification of            the object, an index calculating formula and a coefficient            necessary for calculating the evaluation index value; and        -   determining the evaluation index value using the selected            index calculating formula and coefficient.    -   (17)    -   The object classification system of any one of (14) to (16),        wherein the object in the received color image is a food, and        wherein the characteristic of the object is a freshness or a        nutrient content of the food.    -   (18)    -   The object classification system of any one of (14) to (17),        wherein the processing circuity is further configured to:        -   identify a region of the received color image that includes            the object, wherein the region comprises a region smaller            than the entire image; and        -   classify the object based on the identified region of the            color image.    -   (19)    -   The object classification system of any one of (14) to (18),        further comprising a camera configured to capture the color        image.    -   (20)    -   The object classification system of any one of (14) to (19),        wherein the processing circuity is further configured to        determine the spectrum information associated with the object.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

REFERENCE SIGNS LIST

-   -   301 Training data generating system    -   302 Object recognition system    -   303 Evaluation index presenting system    -   311 Multispectral camera    -   312 RGB camera    -   313 Storage device    -   314 Training data generating processing device    -   315 Output device    -   321 Spectrum information retaining unit    -   322 Recognition processing unit    -   323 Labeling unit    -   324 Evaluation index calculating unit    -   331 Storage device    -   332 Learning tool    -   333 RGB camera    -   334 Output device

The invention claimed is:
 1. An object classification system,comprising: at least one storage device configured to store a trainedobject classifier; and processing circuitry configured to: classify anobject in a received color image using the trained object classifier;determine based, at least in part, on the classification of the objectin the received color image and spectrum information associated with theobject, an evaluation index value for a characteristic of the object,wherein determining the evaluation index for the characteristic of theobject comprises selecting, from a plurality of index calculatingformulas and based at least in part on the classification of the object,an index calculating formula for calculating the evaluation index valueof the object, wherein the plurality of index calculating formulasincludes a second index calculating formula for calculating a secondevaluation index value for a second characteristic of a second object,wherein the characteristic of the object is different from the secondcharacteristic of the second object; and output on a display, anindication of the evaluation index value; wherein training data to trainthe trained object classifier is stored on the at least one storagedevice, the training data including labels associated with storedspectrum information for a plurality of objects.
 2. The objectclassification system of claim 1, wherein the processing circuitry isfurther configured to output on the display, the received color image,and an indication of the classification of the object in the colorimage.
 3. The object classification system of claim 1, wherein theobject in the received color image is a food, and wherein thecharacteristic of the object is a freshness or a nutrient content of thefood.
 4. The object classification system of claim 1, wherein theprocessing circuitry is further configured to: identify a region of thereceived color image that includes the object, wherein the regioncomprises a region smaller than the entire image; and classify theobject based on the identified region of the color image.
 5. The objectclassification system of claim 1, further comprising a camera configuredto capture the color image.
 6. The object classification system of claim1, wherein the processing circuitry is further configured to determinethe spectrum information associated with the object.
 7. The objectclassification system of claim 6, further comprising a multispectralcamera configured to capture a multispectral image, and wherein theprocessing circuitry is configured to determine the spectrum informationassociated with the object based on the captured multispectral image. 8.The object classification system of claim 1, wherein the classificationof the object is different from a second classification of the secondobject.
 9. The object classification system of claim 1, wherein thesecond in the received color image is a food, and wherein thecharacteristic of the object is a freshness or a nutrient content of thefood.